Oct 29 22:39:50.928755 ip-10-0-134-127 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 29 22:39:50.928768 ip-10-0-134-127 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 29 22:39:50.928777 ip-10-0-134-127 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 29 22:39:50.929090 ip-10-0-134-127 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 29 22:40:01.070512 ip-10-0-134-127 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 29 22:40:01.070528 ip-10-0-134-127 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b936c840e43d407ba4d955877095a3fd --
Oct 29 22:42:04.702110 ip-10-0-134-127 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 22:42:05.057906 ip-10-0-134-127 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:42:05.057906 ip-10-0-134-127 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 22:42:05.057906 ip-10-0-134-127 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:42:05.057906 ip-10-0-134-127 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:42:05.060991 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.060914    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:05.060991 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.060990    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:05.060991 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.060995    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.060999    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061002    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061006    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061008    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061011    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061014    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061017    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061020    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061024    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061028    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061032    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061036    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061044    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061047    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061050    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061053    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061055    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061058    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:05.061086 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061061    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061064    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061067    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061070    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061073    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061076    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061078    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061081    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061084    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061087    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061090    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061093    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061097    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061099    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061102    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061105    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061108    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061111    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061113    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:05.061549 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061116    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061120    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061123    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061126    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061129    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061131    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061134    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061137    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061139    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061142    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061145    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061147    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061150    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061152    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061155    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061159    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061162    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061164    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061167    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061169    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:05.062019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061172    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061175    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061177    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061180    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061182    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061185    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061187    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061190    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061193    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061195    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061198    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061200    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061203    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061206    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061208    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061211    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061213    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061216    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061220    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061222    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:05.062533 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061225    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061228    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061230    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061233    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061235    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061238    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061241    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061244    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061247    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061249    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061252    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061255    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061257    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061260    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061263    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061265    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061269    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061272    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061274    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061277    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:05.063019 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061280    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061283    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061285    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061288    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061290    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.061293    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062767    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062774    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062777    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062781    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062784    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062787    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062790    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062793    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062795    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062812    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062814    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062817    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062821    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:05.063487 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062824    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062827    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062829    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062832    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062834    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062837    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062840    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062843    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062847    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062852    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062855    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062858    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062861    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062864    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062867    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062869    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062872    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062875    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062879    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062881    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:05.063966 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062884    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062887    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062889    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062892    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062895    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062898    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062901    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062904    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062906    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062909    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062911    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062915    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062917    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062920    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062923    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062925    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062928    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062930    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062933    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:05.064445 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062935    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062938    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062940    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062943    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062945    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062948    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062951    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062954    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062956    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062959    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062961    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062964    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062967    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062970    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062972    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062975    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062977    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062980    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062982    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062986    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:05.064908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062988    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062991    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062995    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.062999    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063002    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063005    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063008    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063010    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063013    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063015    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063018    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063021    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063023    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063026    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063029    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063032    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063035    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063037    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063040    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:05.065385 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063042    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063045    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063047    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063050    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063053    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063056    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063059    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063061    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063064    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063067    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063069    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063072    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063074    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063077    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063080    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063549    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063554    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063557    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063560    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063563    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:05.065863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063566    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063569    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063571    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063574    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063577    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063580    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063582    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063586    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063590    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063593    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063596    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063599    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063601    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063604    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063607    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063609    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063614    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063617    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063620    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:05.066353 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063622    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063625    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063628    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063630    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063633    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063636    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063639    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063641    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063644    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063647    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063650    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063653    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063655    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063658    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063661    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063663    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063666    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063668    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063671    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:05.066865 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063674    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063676    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063679    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063681    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063684    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063687    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063689    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063692    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063695    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063697    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063701    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063704    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063708    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063712    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063714    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063717    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063720    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063723    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063725    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:05.067359 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063728    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063731    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063734    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063739    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063741    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063744    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063747    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063749    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063752    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063755    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063757    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063760    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063762    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063765    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063767    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063770    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063773    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063775    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063778    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063781    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:05.067908 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063783    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063786    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063788    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063793    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063796    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063814    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063817    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063819    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063823    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063826    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063828    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063831    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063834    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063837    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063840    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063844    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063847    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063850    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063853    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063855    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:05.068443 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063859    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063862    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063864    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.063867    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064772    2575 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064782    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064788    2575 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064793    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064810    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064814    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064818    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064822    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064826    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064829    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064833    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064836    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064840    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064842    2575 flags.go:64] FLAG: --cgroup-root=""
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064845    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064848    2575 flags.go:64] FLAG: --client-ca-file=""
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064851    2575 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064854    2575 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064859    2575 flags.go:64] FLAG: --cluster-domain=""
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064862    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 22:42:05.069036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064865    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064869    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064872    2575 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064884    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064887    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064890    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064894    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064897    2575 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064900    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064903    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064906    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064909    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064919    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064922    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064925    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064928    2575 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064931    2575 flags.go:64] FLAG: --enable-server="true"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064934    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064939    2575 flags.go:64] FLAG: --event-burst="100"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064942    2575 flags.go:64] FLAG: --event-qps="50"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064945    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064948    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064951    2575 flags.go:64] FLAG: --eviction-hard=""
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064955    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 22:42:05.069703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064958    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064962    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064965    2575 flags.go:64] FLAG: --eviction-soft=""
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064969    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064972    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064974    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064977    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064980    2575 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064983    2575 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064986    2575 flags.go:64] FLAG: --feature-gates=""
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064990    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064993    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.064997    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065000    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065004    2575 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065007    2575 flags.go:64] FLAG: --help="false"
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065010    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-134-127.ec2.internal"
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065013    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065016    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065019    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065022    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065026    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065029    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 22:42:05.070396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065032    2575 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065035    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065038    2575 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065041    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065044    2575 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065047    2575 flags.go:64] FLAG: --kube-reserved=""
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065050    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065053    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065056    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065059    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065062    2575 flags.go:64] FLAG: --lock-file=""
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065065    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065068    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065071    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065076    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065079    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065082    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065085    2575 flags.go:64] FLAG: --logging-format="text"
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065088    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065091    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065094    2575 flags.go:64] FLAG: --manifest-url=""
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065097    2575 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065102    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065105    2575 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065109    2575 flags.go:64] FLAG: --max-pods="110"
Oct 29 22:42:05.071028 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065112    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065115    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065118    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065121    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065124    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065127    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065130    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065137    2575 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065140    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065143    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065146    2575 flags.go:64] FLAG: --pod-cidr=""
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065149    2575 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065152    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065155    2575 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065158    2575 flags.go:64] FLAG: --port="10250"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065161    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065164    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a8abc33e4f6cca8c"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065167    2575 flags.go:64] FLAG: --qos-reserved=""
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065170    2575 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065173    2575 flags.go:64] FLAG: --register-node="true"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065176    2575 flags.go:64] FLAG: --register-with-taints=""
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065180    2575 flags.go:64] FLAG: --registry-burst="10"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065183    2575 flags.go:64] FLAG: --registry-qps="5"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065186    2575 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065189    2575 flags.go:64] FLAG: --reserved-memory=""
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065195    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 22:42:05.071656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065198    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065201    2575 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065204    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065207    2575 flags.go:64] FLAG: --runonce="false"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065210    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065214    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065217    2575 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065221    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065224    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065227    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065231    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065234    2575 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065237    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065240    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065243    2575 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065246    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065249    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065252    2575 flags.go:64] FLAG: --system-cgroups=""
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065255    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065260    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065263    2575 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065266    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065271    2575 flags.go:64] FLAG: --tls-min-version=""
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065274    2575 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065277    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 22:42:05.072299 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065280    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065282    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065285    2575 flags.go:64] FLAG: --v="2"
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065290    2575 flags.go:64] FLAG: --version="false"
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065294    2575 flags.go:64] FLAG: --vmodule=""
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065298    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065301    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065430    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065434    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065437    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065440    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065443    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065446    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065449    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065453    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065456    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065459    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065462    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065465    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065468    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065470    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:05.072893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065473    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065476    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065480    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065484    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065487    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065490    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065493    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065496    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065498    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065501    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065504    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065506    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065509    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065512    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065514    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065517    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065520    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065522    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065527    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:05.073435 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065530    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065533    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065535    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065538    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065540    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065543    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065548    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065551    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065553    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065556    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065559    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065561    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065564    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065566    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065569    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065572    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065574    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065577    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065580    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:05.074167 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065582    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065585    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065588    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065590    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065593    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065596    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065598    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065601    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065603    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065606    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065609    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065611    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065615    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065618    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065620    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065623    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065626    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065628    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065631    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065636    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:05.074875 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065638    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065641    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065644    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065646    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065649    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065652    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065655    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065657    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065660    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065663    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065665    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065668    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065670    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065673    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065676    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065678    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065681    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065684    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065687    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065689    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:05.075466 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065692    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:05.076006 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065694    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:05.076006 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065697    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:05.076006 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065699    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:05.076006 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065703    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:05.076006 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065706    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:05.076006 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065709    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:05.076006 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065711    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:05.076006 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065714    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:05.076006 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065717    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:05.076006 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065720    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:05.076006 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065725    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:05.076006 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065729    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:05.076006 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065732    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:05.076006 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.065796    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:42:05.076006 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065852    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065857    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065860    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065863    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065866    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065869    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065872    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065875    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065878    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065881    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065883    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065886    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065888    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065891    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065894    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065897    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065900    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065903    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065905    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065908    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:05.076391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065911    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065916    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065919    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065922    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065924    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065927    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065930    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065933    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065935    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065940    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065946    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065950    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065953    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065957    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065959    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065962    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065965    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065968    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065970    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:05.076961 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065973    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065976    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065979    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065982    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065984    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065987    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065990    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065993    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065995    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.065998    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066001    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066004    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066008    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066011    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066017    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066020    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066023    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066026    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066028    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:05.077418 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066031    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066034    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066037    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066041    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066044    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066047    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066050    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066053    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066056    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066058    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066061    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066064    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066066    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066069    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066072    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066074    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066077    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066080    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066082    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:05.077946 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066085    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066088    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066090    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066093    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066096    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066098    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066101    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066104    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066109    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066112    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066115    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066118    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066120    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066123    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066126    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066128    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066133    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066135    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066138    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066141    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:05.078399 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066143    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066146    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066149    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066151    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066155    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066158    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066161    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066163    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.066166    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.072910    2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.072926    2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.072941    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.072949    2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073009    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073015    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073018    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073021    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073024    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073027    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073030    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:05.078893 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073033    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073035    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073038    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073042    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073052    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073056    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073059    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073062    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073064    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073068    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073072    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073075    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073079    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073081    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073084    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073087    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073090    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073093    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073095    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:05.079429 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073098    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073101    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073103    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073113    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073116    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073119    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073122    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073124    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073127    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073129    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073132    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073135    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073137    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073140    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073143    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073145    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073148    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073150    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073153    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:05.079947 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073156    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073158    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073161    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073164    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073167    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073169    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073172    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073174    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073177    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073180    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073182    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073185    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073187    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073190    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073193    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073196    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073199    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073208    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073211    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073213    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:05.080410 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073216    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073219    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073221    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073224    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073227    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073229    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073232    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073235    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073237    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073240    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073243    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073245    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073248    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073250    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073253    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073256    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073258    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073261    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073263    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:05.080950 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073266    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073269    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073271    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073274    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073276    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073279    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073282    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073285    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073287    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073290    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073293    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073301    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073304    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073306    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073309    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073312    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073314    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073317    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073319    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073322    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:05.081391 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073324    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:05.081905 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.073327    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:05.081905 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.073354    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:42:05.081905 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.074975    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:05.081905 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.074989    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:05.081905 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.074993    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:05.081905 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.074996    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:05.081905 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.074999    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:05.081905 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075002    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:05.081905 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075005    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:05.081905 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075008    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:05.081905 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075011    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:05.081905 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075014    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:05.081905 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075017    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:05.081905 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075020    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:05.081905 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075022    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:05.081905 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075025    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075028    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075031    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075034    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075036    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075040    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075043    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075046    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075049    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075051    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075054    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075057    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075059    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075062    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075065    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075067    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075070    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075073    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075075    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:05.082331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075078    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075081    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075084    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075087    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075089    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075092    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075095    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075097    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075100    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075103    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075105    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075108    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075110    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075113    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075116    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075119    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075121    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075124    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075128    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:05.082919 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075131    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075134    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075137    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075139    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075142    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075145    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075147    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075150    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075152    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075156    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075160    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075164    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075167    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075169    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075173    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075177    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075180    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075183    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075185    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:05.083364 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075188    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075191    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075193    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075196    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075199    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075201    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075204    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075206    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075209    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075212    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075215    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075217    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075220    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075224    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075226    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075229    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075232    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075235    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075237    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075240    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:05.083863 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075242    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:05.084350 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075245    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:05.084350 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075248    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:05.084350 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075250    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:05.084350 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075253    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:05.084350 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075256    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:05.084350 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075258    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:05.084350 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075261    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:05.084350 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075264    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:05.084350 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075266    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:05.084350 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075269    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:05.084350 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075271    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:05.084350 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075274    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:05.084350 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075277    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:05.084350 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075279    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:05.084350 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075282    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:05.084350 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.075310    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075351    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075357    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075361    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075364    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075367    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075369    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075372    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075375    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075379    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075384    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075387    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075390    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075393    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075397    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075400    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075404    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075406    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075409    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:05.084725 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075412    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075414    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075417    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075420    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075422    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075425    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075427    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075430    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075433    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075436    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075438    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075441    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075443    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075446    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075449    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075452    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075454    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075457    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075460    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075463    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:05.085305 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075465    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075468    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075471    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075473    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075477    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075480    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075482    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075485    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075488    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075490    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075493    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075496    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075498    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075501    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075503    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075506    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075508    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075511    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075514    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075516    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:05.085827 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075519    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075521    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075524    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075528    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075530    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075533    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075536    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075538    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075541    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075544    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075547    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075549    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075552    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075554    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075557    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075560    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075562    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075565    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075568    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075570    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:05.086331 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075573    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075576    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075578    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075581    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075583    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075586    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075588    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075591    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075593    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075596    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075599    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075601    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075604    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075610    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075613    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075615    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075618    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075621    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075623    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:05.086822 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075628    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:05.087266 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075630    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:05.087266 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075633    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:05.087266 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075636    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:05.087266 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075638    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:05.087266 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075641    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:05.087266 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075644    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:05.087266 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075646    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:05.087266 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:05.075649    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:05.087266 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.075706    2575 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 22:42:05.087266 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.077698    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 29 22:42:05.087266 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.079021    2575 server.go:1014] "Starting client certificate rotation"
Oct 29 22:42:05.087266 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.079111    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:42:05.087266 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.079141    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:42:05.087266 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.080842    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:42:05.087266 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.081953    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:42:05.092773 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.092758    2575 log.go:25] "Validated CRI v1 runtime API"
Oct 29 22:42:05.095676 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.095660    2575 log.go:25] "Validated CRI v1 image API"
Oct 29 22:42:05.096597 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.096574    2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 22:42:05.103426 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.103406    2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 80e69dd1-2d10-4b80-adfc-3e8e869fafd9:/dev/nvme0n1p4 9226ec9e-39c2-487f-b83e-e498690ea092:/dev/nvme0n1p3]
Oct 29 22:42:05.103489 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.103425    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 29 22:42:05.111076 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.110962    2575 manager.go:217] Machine: {Timestamp:2026-10-29 22:42:05.109581786 +0000 UTC m=+0.304403854 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099868 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26c30ad3e6218ba2bd0075085c8ebc SystemUUID:ec26c30a-d3e6-218b-a2bd-0075085c8ebc BootID:b936c840-e43d-407b-a4d9-55877095a3fd Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:da:3c:e2:17 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:da:3c:e2:17 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:1e:8b:0e:51:18 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 22:42:05.111076 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.111074    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 22:42:05.111183 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.111146    2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 22:42:05.113214 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.113192    2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 22:42:05.113336 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.113216    2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-127.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 22:42:05.113383 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.113339    2575 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 22:42:05.113383 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.113347    2575 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 22:42:05.113383 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.113380    2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:42:05.113458 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.113399    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:42:05.114147 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.114136    2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 22:42:05.117850 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.117839    2575 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 22:42:05.117963 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.117954    2575 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 22:42:05.118535 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.118527    2575 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 22:42:05.118563 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.118543    2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 22:42:05.118563 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.118553    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 22:42:05.118620 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.118575    2575 kubelet.go:405] "Adding apiserver pod source"
Oct 29 22:42:05.118620 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.118584    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 22:42:05.118984 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.118971    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:42:05.122371 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.122347    2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 22:42:05.127782 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.127769    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:42:05.128687 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.128675    2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 22:42:05.128723 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.128705    2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 22:42:05.128879 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.128869    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 22:42:05.128908 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.128886    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 22:42:05.128908 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.128892    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 22:42:05.128908 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.128898    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 22:42:05.128908 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.128904    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 22:42:05.129014 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.128912    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 22:42:05.129508 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.129499    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 22:42:05.129535 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.129509    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 22:42:05.129535 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.129516    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 22:42:05.129535 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.129522    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 22:42:05.129535 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.129531    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 22:42:05.129655 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.129539    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 22:42:05.130073 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.130064    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 22:42:05.130111 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.130074    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 22:42:05.133671 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.133657    2575 server.go:1263] "Started kubelet"
Oct 29 22:42:05.133831 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.133790    2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 22:42:05.133903 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.133785    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 22:42:05.133903 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.133882    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 22:42:05.134387 ip-10-0-134-127 systemd[1]: Started Kubernetes Kubelet.
Oct 29 22:42:05.134855 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.134840    2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 22:42:05.136341 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.136327    2575 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 22:42:05.139327 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.139292    2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 29 22:42:05.140565 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.140546    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 22:42:05.141022 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.141005    2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 22:42:05.141299 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.141138    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-127.ec2.internal\" not found"
Oct 29 22:42:05.141663 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.141647    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 22:42:05.141735 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.141648    2575 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 22:42:05.141735 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.141680    2575 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 22:42:05.141846 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.141765    2575 factory.go:55] Registering systemd factory
Oct 29 22:42:05.141846 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.141776    2575 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 22:42:05.141846 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.141793    2575 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 22:42:05.141846 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.141780    2575 factory.go:223] Registration of the systemd container factory successfully
Oct 29 22:42:05.142058 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.142046    2575 factory.go:153] Registering CRI-O factory
Oct 29 22:42:05.142113 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.142061    2575 factory.go:223] Registration of the crio container factory successfully
Oct 29 22:42:05.142163 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.142118    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 22:42:05.142163 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.142137    2575 factory.go:103] Registering Raw factory
Oct 29 22:42:05.142163 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.142154    2575 manager.go:1199] Started watching for new ooms in manager
Oct 29 22:42:05.143036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.143014    2575 manager.go:319] Starting recovery of all containers
Oct 29 22:42:05.153665 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.153643    2575 manager.go:324] Recovery completed
Oct 29 22:42:05.159444 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.159369    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:05.161229 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.161207    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:05.161292 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.161241    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:05.161292 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.161251    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:05.161813 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.161783    2575 cpu_manager.go:226] "Starting" policy="none"
Oct 29 22:42:05.161913 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.161820    2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 22:42:05.161913 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.161841    2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 22:42:05.161913 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.161796    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:05.162605 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.162591    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:05.162650 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.162615    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:05.162650 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.162627    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:05.164126 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.164113    2575 policy_none.go:50] "Start"
Oct 29 22:42:05.164193 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.164130    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 22:42:05.164193 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.164140    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 22:42:05.166134 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.166124    2575 policy_none.go:44] "Start"
Oct 29 22:42:05.170827 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.170791    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 22:42:05.172487 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.172459    2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-127.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 29 22:42:05.172487 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.172462    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-127.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 29 22:42:05.173370 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.172460    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-127.ec2.internal.18e32154e56c2d77  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-127.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-127.ec2.internal,},FirstTimestamp:2026-10-29 22:42:05.133630839 +0000 UTC m=+0.328452905,LastTimestamp:2026-10-29 22:42:05.133630839 +0000 UTC m=+0.328452905,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-127.ec2.internal,}"
Oct 29 22:42:05.179601 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.179580    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m9btz"
Oct 29 22:42:05.187367 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.187277    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-127.ec2.internal.18e32154e7114712  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-127.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-127.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-127.ec2.internal,},FirstTimestamp:2026-10-29 22:42:05.16122805 +0000 UTC m=+0.356050116,LastTimestamp:2026-10-29 22:42:05.16122805 +0000 UTC m=+0.356050116,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-127.ec2.internal,}"
Oct 29 22:42:05.187884 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.187862    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-127.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 29 22:42:05.187884 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.187862    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 29 22:42:05.191983 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.191966    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m9btz"
Oct 29 22:42:05.194387 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.194361    2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 22:42:05.196994 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.196980    2575 manager.go:341] "Starting Device Plugin manager"
Oct 29 22:42:05.197075 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.197012    2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 22:42:05.197075 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.197020    2575 server.go:87] "Starting device plugin registration server"
Oct 29 22:42:05.197257 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.197245    2575 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 22:42:05.197293 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.197261    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 22:42:05.197369 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.197355    2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 22:42:05.197461 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.197441    2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 22:42:05.197461 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.197456    2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 22:42:05.198056 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.198027    2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 22:42:05.198127 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.198091    2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-127.ec2.internal\" not found"
Oct 29 22:42:05.248945 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.247929    2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:05.262954 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.262925    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 22:42:05.264333 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.264312    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 22:42:05.264333 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.264331    2575 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 22:42:05.264444 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.264354    2575 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 22:42:05.264444 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.264393    2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 29 22:42:05.278715 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.278700    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:05.297966 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.297951    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:05.298815 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.298791    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:05.298884 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.298832    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:05.298884 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.298843    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:05.298884 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.298866    2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-127.ec2.internal"
Oct 29 22:42:05.316016 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.315981    2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-127.ec2.internal"
Oct 29 22:42:05.316016 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.316000    2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-127.ec2.internal\": node \"ip-10-0-134-127.ec2.internal\" not found"
Oct 29 22:42:05.363378 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.363359    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-127.ec2.internal\" not found"
Oct 29 22:42:05.365457 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.365442    2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"]
Oct 29 22:42:05.365507 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.365488    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:05.367130 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.367116    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:05.367202 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.367142    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:05.367202 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.367157    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:05.369312 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.369299    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:05.369432 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.369418    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal"
Oct 29 22:42:05.369474 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.369444    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:05.370186 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.370163    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:05.370186 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.370173    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:05.370276 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.370187    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:05.370276 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.370193    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:05.370276 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.370200    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:05.370358 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.370202    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:05.372229 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.372216    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:42:05.372286 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.372240    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:05.373321 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.373308    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:05.373394 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.373332    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:05.373394 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.373346    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:05.387228 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.387207    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-127.ec2.internal\" not found" node="ip-10-0-134-127.ec2.internal"
Oct 29 22:42:05.393077 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.393059    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-127.ec2.internal\" not found" node="ip-10-0-134-127.ec2.internal"
Oct 29 22:42:05.444211 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.444184    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/404a57dfae016c255d65ef53a28af2de-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal\" (UID: \"404a57dfae016c255d65ef53a28af2de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:42:05.444322 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.444214    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a47eb3de0ec2692f9f0b4d5a8aa019c1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-127.ec2.internal\" (UID: \"a47eb3de0ec2692f9f0b4d5a8aa019c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal"
Oct 29 22:42:05.444322 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.444230    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/404a57dfae016c255d65ef53a28af2de-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal\" (UID: \"404a57dfae016c255d65ef53a28af2de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:42:05.464340 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.464318    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-127.ec2.internal\" not found"
Oct 29 22:42:05.545035 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.545009    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/404a57dfae016c255d65ef53a28af2de-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal\" (UID: \"404a57dfae016c255d65ef53a28af2de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:42:05.545139 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.545047    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/404a57dfae016c255d65ef53a28af2de-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal\" (UID: \"404a57dfae016c255d65ef53a28af2de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:42:05.545139 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.545090    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/404a57dfae016c255d65ef53a28af2de-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal\" (UID: \"404a57dfae016c255d65ef53a28af2de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:42:05.545139 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.545123    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a47eb3de0ec2692f9f0b4d5a8aa019c1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-127.ec2.internal\" (UID: \"a47eb3de0ec2692f9f0b4d5a8aa019c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal"
Oct 29 22:42:05.545295 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.545147    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a47eb3de0ec2692f9f0b4d5a8aa019c1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-127.ec2.internal\" (UID: \"a47eb3de0ec2692f9f0b4d5a8aa019c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal"
Oct 29 22:42:05.545295 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.545168    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/404a57dfae016c255d65ef53a28af2de-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal\" (UID: \"404a57dfae016c255d65ef53a28af2de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:42:05.564881 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.564862    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-127.ec2.internal\" not found"
Oct 29 22:42:05.664986 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.664959    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-127.ec2.internal\" not found"
Oct 29 22:42:05.690642 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.690614    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal"
Oct 29 22:42:05.697593 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:05.697472    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:42:05.765588 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.765550    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-127.ec2.internal\" not found"
Oct 29 22:42:05.866039 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.866012    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-127.ec2.internal\" not found"
Oct 29 22:42:05.966514 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:05.966452    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-127.ec2.internal\" not found"
Oct 29 22:42:06.067065 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:06.067035    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-127.ec2.internal\" not found"
Oct 29 22:42:06.078232 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:06.078213    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 29 22:42:06.078360 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:06.078340    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 29 22:42:06.078424 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:06.078394    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 29 22:42:06.168378 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:06.168345    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-127.ec2.internal\" not found"
Oct 29 22:42:06.196417 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:06.196376    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:37:05 +0000 UTC" deadline="2028-04-09 21:03:26.983323735 +0000 UTC"
Oct 29 22:42:06.196417 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:06.196409    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12670h21m20.786917948s"
Oct 29 22:42:06.261428 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:06.261380    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404a57dfae016c255d65ef53a28af2de.slice/crio-58676504ac4c1ffb5167915e7395604a0c201844b2e26d83f32c3a64cc07a6f2 WatchSource:0}: Error finding container 58676504ac4c1ffb5167915e7395604a0c201844b2e26d83f32c3a64cc07a6f2: Status 404 returned error can't find the container with id 58676504ac4c1ffb5167915e7395604a0c201844b2e26d83f32c3a64cc07a6f2
Oct 29 22:42:06.266564 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:06.266545    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:42:06.267082 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:06.267064    2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:06.267718 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:06.267648    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal" event={"ID":"404a57dfae016c255d65ef53a28af2de","Type":"ContainerStarted","Data":"58676504ac4c1ffb5167915e7395604a0c201844b2e26d83f32c3a64cc07a6f2"}
Oct 29 22:42:06.271581 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:06.271559    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47eb3de0ec2692f9f0b4d5a8aa019c1.slice/crio-78f01703269066ee6b5be698312e3856eee8652f2fbbf9dcf4f145eb25743e8a WatchSource:0}: Error finding container 78f01703269066ee6b5be698312e3856eee8652f2fbbf9dcf4f145eb25743e8a: Status 404 returned error can't find the container with id 78f01703269066ee6b5be698312e3856eee8652f2fbbf9dcf4f145eb25743e8a
Oct 29 22:42:06.343282 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:06.343259    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal"
Oct 29 22:42:06.371962 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:06.371936    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:42:06.372074 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:06.372046    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:42:06.398794 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:06.398771    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:42:06.493544 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:06.493515    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:07.119347 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.119312    2575 apiserver.go:52] "Watching apiserver"
Oct 29 22:42:07.142574 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.142533    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 29 22:42:07.172986 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.172955    2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 22:42:07.173381 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.173357    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-qmfpg","kube-system/konnectivity-agent-5jn2n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666","openshift-multus/multus-8dtlv","openshift-network-diagnostics/network-check-target-lqjl6","openshift-network-operator/iptables-alerter-2q4dp","openshift-ovn-kubernetes/ovnkube-node-4xc2d","kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal","openshift-cluster-node-tuning-operator/tuned-4bgs7","openshift-dns/node-resolver-q8pg5","openshift-image-registry/node-ca-n27f7","openshift-multus/multus-additional-cni-plugins-mlsbk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal","openshift-multus/network-metrics-daemon-prbcb"]
Oct 29 22:42:07.176842 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.176796    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.179133 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.179108    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n27f7"
Oct 29 22:42:07.181693 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.181667    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.184151 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.184123    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.186610 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.186586    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:07.186701 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.186681    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:07.189230 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.189210    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2q4dp"
Oct 29 22:42:07.191600 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.191581    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:07.191711 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.191650    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:07.194147 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.194129    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.196200 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.196014    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 22:42:07.196200 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.196014    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rpfjl\""
Oct 29 22:42:07.197632 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.197609    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5jn2n"
Oct 29 22:42:07.200189 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.200168    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 22:42:07.200293 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.200278    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 22:42:07.200340 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.200306    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 22:42:07.200413 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.200400    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 22:42:07.200482 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.200470    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 22:42:07.200760 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.200742    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 22:42:07.204127 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.204110    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 22:42:07.204556 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.204340    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 22:42:07.204556 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.204367    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 22:42:07.204556 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.204411    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 22:42:07.204556 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.204465    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 22:42:07.204850 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.204597    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 22:42:07.204850 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.204694    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 22:42:07.204961 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.204886    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 22:42:07.205582 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.205490    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q8pg5"
Oct 29 22:42:07.208011 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.207978    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:07.208224 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.208193    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:07.210664 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.210644    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.217723 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.217701    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 22:42:07.217815 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.217769    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-w2tv8"
Oct 29 22:42:07.242852 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.242629    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 22:42:07.255658 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.255616    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/752e574a-9cd1-40a1-a65e-7b2aabd4434e-tmp\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.255742 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.255655    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-multus-socket-dir-parent\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.255742 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.255684    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50192cd-cb7b-4b3e-978f-661093207b81-tmp-dir\") pod \"node-resolver-q8pg5\" (UID: \"e50192cd-cb7b-4b3e-978f-661093207b81\") " pod="openshift-dns/node-resolver-q8pg5"
Oct 29 22:42:07.255742 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.255710    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-log-socket\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.255742 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.255732    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c68dp\" (UniqueName: \"kubernetes.io/projected/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-kube-api-access-c68dp\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.255995 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.255746    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-run\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.255995 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.255759    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da8029d1-50de-4612-adce-3a3f582c74d8-host\") pod \"node-ca-n27f7\" (UID: \"da8029d1-50de-4612-adce-3a3f582c74d8\") " pod="openshift-image-registry/node-ca-n27f7"
Oct 29 22:42:07.255995 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.255774    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-cnibin\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.255995 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.255791    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-multus-conf-dir\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.255995 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.255836    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d6fdbfe-0357-4139-a034-28faec681123-multus-daemon-config\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.255995 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.255856    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-etc-openvswitch\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.255995 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.255871    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-device-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.255995 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.255909    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/98c4af2a-2166-4fa0-99d0-662be28b6899-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.255995 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.255975    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-kubelet\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.256468 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256005    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs\") pod \"network-metrics-daemon-prbcb\" (UID: \"d7f074e8-1737-4f5c-99cd-928d9c565c17\") " pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:07.256468 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256079    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-host\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.256468 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256114    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-etc-selinux\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.256468 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256190    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-hostroot\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.256468 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256236    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0d1b5f20-523b-4c79-a942-b0dfce8f64f9-iptables-alerter-script\") pod \"iptables-alerter-2q4dp\" (UID: \"0d1b5f20-523b-4c79-a942-b0dfce8f64f9\") " pod="openshift-network-operator/iptables-alerter-2q4dp"
Oct 29 22:42:07.256468 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256286    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98c4af2a-2166-4fa0-99d0-662be28b6899-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.256468 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256347    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.256468 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256385    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:07.256468 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256409    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-tuned\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.256468 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256426    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98c4af2a-2166-4fa0-99d0-662be28b6899-cnibin\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.256978 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256474    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb7k9\" (UniqueName: \"kubernetes.io/projected/98c4af2a-2166-4fa0-99d0-662be28b6899-kube-api-access-gb7k9\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.256978 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256501    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-ovnkube-script-lib\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.256978 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256524    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d6fdbfe-0357-4139-a034-28faec681123-cni-binary-copy\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.256978 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256550    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-var-lib-cni-bin\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.256978 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256576    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98c4af2a-2166-4fa0-99d0-662be28b6899-cni-binary-copy\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.256978 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256602    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ndcg\" (UniqueName: \"kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg\") pod \"network-check-target-lqjl6\" (UID: \"8ba702dc-08af-46e3-8235-61da0c303efd\") " pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:07.256978 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256626    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fea00aae-5dd9-48ef-b7d4-af08eed1481d-konnectivity-ca\") pod \"konnectivity-agent-5jn2n\" (UID: \"fea00aae-5dd9-48ef-b7d4-af08eed1481d\") " pod="kube-system/konnectivity-agent-5jn2n"
Oct 29 22:42:07.256978 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256657    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-sysconfig\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.256978 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256695    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-systemd\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.256978 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256723    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-var-lib-kubelet\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.256978 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256776    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.256978 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256802    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/98c4af2a-2166-4fa0-99d0-662be28b6899-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.256978 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256840    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e50192cd-cb7b-4b3e-978f-661093207b81-hosts-file\") pod \"node-resolver-q8pg5\" (UID: \"e50192cd-cb7b-4b3e-978f-661093207b81\") " pod="openshift-dns/node-resolver-q8pg5"
Oct 29 22:42:07.256978 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256856    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da8029d1-50de-4612-adce-3a3f582c74d8-serviceca\") pod \"node-ca-n27f7\" (UID: \"da8029d1-50de-4612-adce-3a3f582c74d8\") " pod="openshift-image-registry/node-ca-n27f7"
Oct 29 22:42:07.256978 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256875    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-system-cni-dir\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.256978 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256894    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-run-k8s-cni-cncf-io\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.257783 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256944    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-run-openvswitch\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.257783 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.256976    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.257783 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257017    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fea00aae-5dd9-48ef-b7d4-af08eed1481d-agent-certs\") pod \"konnectivity-agent-5jn2n\" (UID: \"fea00aae-5dd9-48ef-b7d4-af08eed1481d\") " pod="kube-system/konnectivity-agent-5jn2n"
Oct 29 22:42:07.257783 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257050    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6edb8e5f-e185-48be-8fed-5ac6899134fa-kubelet-config\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:07.257783 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257075    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-sys\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.257783 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257123    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d1b5f20-523b-4c79-a942-b0dfce8f64f9-host-slash\") pod \"iptables-alerter-2q4dp\" (UID: \"0d1b5f20-523b-4c79-a942-b0dfce8f64f9\") " pod="openshift-network-operator/iptables-alerter-2q4dp"
Oct 29 22:42:07.257783 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257155    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-run-netns\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.257783 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257173    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6edb8e5f-e185-48be-8fed-5ac6899134fa-dbus\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:07.257783 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257193    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6q45\" (UniqueName: \"kubernetes.io/projected/765947ed-ccf0-4296-aac8-eace5ce6de20-kube-api-access-p6q45\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.257783 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257217    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-multus-cni-dir\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.257783 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257238    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-run-multus-certs\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.257783 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257256    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98c4af2a-2166-4fa0-99d0-662be28b6899-system-cni-dir\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.257783 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257288    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-node-log\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.257783 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257347    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-var-lib-cni-multus\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.257783 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257366    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-var-lib-kubelet\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.257783 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257381    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6ncm\" (UniqueName: \"kubernetes.io/projected/e50192cd-cb7b-4b3e-978f-661093207b81-kube-api-access-l6ncm\") pod \"node-resolver-q8pg5\" (UID: \"e50192cd-cb7b-4b3e-978f-661093207b81\") " pod="openshift-dns/node-resolver-q8pg5"
Oct 29 22:42:07.258425 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257398    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-cni-netd\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.258425 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257432    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-ovn-node-metrics-cert\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.258425 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257468    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-sysctl-conf\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.258425 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257496    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-socket-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.258425 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257529    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-run-systemd\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.258425 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257560    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-sysctl-d\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.258425 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257591    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-lib-modules\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.258425 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257617    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f8gp\" (UniqueName: \"kubernetes.io/projected/da8029d1-50de-4612-adce-3a3f582c74d8-kube-api-access-9f8gp\") pod \"node-ca-n27f7\" (UID: \"da8029d1-50de-4612-adce-3a3f582c74d8\") " pod="openshift-image-registry/node-ca-n27f7"
Oct 29 22:42:07.258425 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257664    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-registration-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.258425 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257689    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-sys-fs\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.258425 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257714    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-run-netns\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.258425 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257740    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-slash\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.258425 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257769    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-cni-bin\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.258425 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257792    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmwh\" (UniqueName: \"kubernetes.io/projected/d7f074e8-1737-4f5c-99cd-928d9c565c17-kube-api-access-6hmwh\") pod \"network-metrics-daemon-prbcb\" (UID: \"d7f074e8-1737-4f5c-99cd-928d9c565c17\") " pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:07.258425 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257840    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qckd7\" (UniqueName: \"kubernetes.io/projected/752e574a-9cd1-40a1-a65e-7b2aabd4434e-kube-api-access-qckd7\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.258425 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257860    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srpcs\" (UniqueName: \"kubernetes.io/projected/7d6fdbfe-0357-4139-a034-28faec681123-kube-api-access-srpcs\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.259010 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257882    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-env-overrides\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.259010 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257904    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-modprobe-d\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.259010 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257928    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l295\" (UniqueName: \"kubernetes.io/projected/0d1b5f20-523b-4c79-a942-b0dfce8f64f9-kube-api-access-4l295\") pod \"iptables-alerter-2q4dp\" (UID: \"0d1b5f20-523b-4c79-a942-b0dfce8f64f9\") " pod="openshift-network-operator/iptables-alerter-2q4dp"
Oct 29 22:42:07.259010 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257951    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-var-lib-openvswitch\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.259010 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.257977    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-run-ovn\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.259010 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.258002    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-ovnkube-config\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.259010 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.258052    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-etc-kubernetes\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.259010 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.258088    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-systemd-units\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.259010 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.258125    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-kubernetes\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.271303 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.271273    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal" event={"ID":"a47eb3de0ec2692f9f0b4d5a8aa019c1","Type":"ContainerStarted","Data":"78f01703269066ee6b5be698312e3856eee8652f2fbbf9dcf4f145eb25743e8a"}
Oct 29 22:42:07.304640 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.304615    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 22:42:07.305076 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.305054    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-phcjr\""
Oct 29 22:42:07.305171 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.305109    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 22:42:07.305171 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.305144    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n7jjw\""
Oct 29 22:42:07.305171 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.305166    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h2plp\""
Oct 29 22:42:07.305475 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.305395    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-fchfg\""
Oct 29 22:42:07.305475 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.305439    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m52c5\""
Oct 29 22:42:07.305475 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.305458    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tj892\""
Oct 29 22:42:07.305475 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.305460    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-28brx\""
Oct 29 22:42:07.305670 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.305441    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xjmd7\""
Oct 29 22:42:07.308642 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.308620    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:42:07.308740 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.308669    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 22:42:07.308915 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.308898    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 22:42:07.308986 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.308899    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:42:07.308986 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.308920    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 22:42:07.308986 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.308947    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 22:42:07.308986 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.308948    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 22:42:07.309197 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.308996    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 22:42:07.309197 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.309042    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 22:42:07.309197 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.309086    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 22:42:07.348855 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.348827    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:07.363159 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363125    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-run-openvswitch\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.363298 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363179    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.363298 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363207    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fea00aae-5dd9-48ef-b7d4-af08eed1481d-agent-certs\") pod \"konnectivity-agent-5jn2n\" (UID: \"fea00aae-5dd9-48ef-b7d4-af08eed1481d\") " pod="kube-system/konnectivity-agent-5jn2n"
Oct 29 22:42:07.363298 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363238    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6edb8e5f-e185-48be-8fed-5ac6899134fa-kubelet-config\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:07.363298 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363273    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-sys\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.363298 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363273    2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:07.363597 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363303    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d1b5f20-523b-4c79-a942-b0dfce8f64f9-host-slash\") pod \"iptables-alerter-2q4dp\" (UID: \"0d1b5f20-523b-4c79-a942-b0dfce8f64f9\") " pod="openshift-network-operator/iptables-alerter-2q4dp"
Oct 29 22:42:07.363597 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363335    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-run-netns\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.363597 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363361    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6edb8e5f-e185-48be-8fed-5ac6899134fa-dbus\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:07.363597 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363392    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6q45\" (UniqueName: \"kubernetes.io/projected/765947ed-ccf0-4296-aac8-eace5ce6de20-kube-api-access-p6q45\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.363597 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363423    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-multus-cni-dir\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.363597 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-run-multus-certs\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.363597 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363485    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98c4af2a-2166-4fa0-99d0-662be28b6899-system-cni-dir\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.363597 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363510    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-node-log\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.363597 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363524    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-sys\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.363597 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-var-lib-cni-multus\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.363597 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363584    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-run-openvswitch\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.364181 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363625    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-var-lib-cni-multus\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.364181 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363630    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-var-lib-kubelet\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.364181 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363673    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.364181 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363683    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-run-netns\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.364181 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363708    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-var-lib-kubelet\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.364181 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363889    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-multus-cni-dir\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.364181 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.363938    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-node-log\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.364181 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.364134    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-run-multus-certs\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.364613 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.364201    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98c4af2a-2166-4fa0-99d0-662be28b6899-system-cni-dir\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.364613 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.364248    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6ncm\" (UniqueName: \"kubernetes.io/projected/e50192cd-cb7b-4b3e-978f-661093207b81-kube-api-access-l6ncm\") pod \"node-resolver-q8pg5\" (UID: \"e50192cd-cb7b-4b3e-978f-661093207b81\") " pod="openshift-dns/node-resolver-q8pg5"
Oct 29 22:42:07.364613 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.364255    2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 22:42:07.364613 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.364285    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-cni-netd\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.364856 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.364637    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6edb8e5f-e185-48be-8fed-5ac6899134fa-kubelet-config\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:07.364856 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.364315    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-ovn-node-metrics-cert\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.364972 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.364868    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-sysctl-conf\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.364972 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.364934    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6edb8e5f-e185-48be-8fed-5ac6899134fa-dbus\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:07.365065 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.364997    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-socket-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.365065 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365034    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-run-systemd\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.365170 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365062    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-sysctl-d\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.365170 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365100    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-socket-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.365170 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365143    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d1b5f20-523b-4c79-a942-b0dfce8f64f9-host-slash\") pod \"iptables-alerter-2q4dp\" (UID: \"0d1b5f20-523b-4c79-a942-b0dfce8f64f9\") " pod="openshift-network-operator/iptables-alerter-2q4dp"
Oct 29 22:42:07.365170 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365144    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-lib-modules\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.365389 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365171    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-cni-netd\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.365389 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365189    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f8gp\" (UniqueName: \"kubernetes.io/projected/da8029d1-50de-4612-adce-3a3f582c74d8-kube-api-access-9f8gp\") pod \"node-ca-n27f7\" (UID: \"da8029d1-50de-4612-adce-3a3f582c74d8\") " pod="openshift-image-registry/node-ca-n27f7"
Oct 29 22:42:07.365389 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365278    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-run-systemd\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.365389 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365290    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-sysctl-d\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.365389 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365380    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-lib-modules\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.365641 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365403    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-sysctl-conf\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.365641 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365424    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-registration-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.365641 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365472    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-registration-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.365641 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365474    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-sys-fs\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.365641 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365518    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-run-netns\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.365641 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365556    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-slash\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.365641 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365598    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-cni-bin\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.365641 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365632    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmwh\" (UniqueName: \"kubernetes.io/projected/d7f074e8-1737-4f5c-99cd-928d9c565c17-kube-api-access-6hmwh\") pod \"network-metrics-daemon-prbcb\" (UID: \"d7f074e8-1737-4f5c-99cd-928d9c565c17\") " pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:07.366058 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365664    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qckd7\" (UniqueName: \"kubernetes.io/projected/752e574a-9cd1-40a1-a65e-7b2aabd4434e-kube-api-access-qckd7\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.366058 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365694    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srpcs\" (UniqueName: \"kubernetes.io/projected/7d6fdbfe-0357-4139-a034-28faec681123-kube-api-access-srpcs\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.366058 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365775    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-env-overrides\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.366058 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365808    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-modprobe-d\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.366058 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365838    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l295\" (UniqueName: \"kubernetes.io/projected/0d1b5f20-523b-4c79-a942-b0dfce8f64f9-kube-api-access-4l295\") pod \"iptables-alerter-2q4dp\" (UID: \"0d1b5f20-523b-4c79-a942-b0dfce8f64f9\") " pod="openshift-network-operator/iptables-alerter-2q4dp"
Oct 29 22:42:07.366058 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365864    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-var-lib-openvswitch\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.366058 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365914    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-run-ovn\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.366058 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.365946    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-ovnkube-config\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.366444 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.366067    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-etc-kubernetes\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.366444 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.366101    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-systemd-units\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.366444 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.366131    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-kubernetes\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.366444 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.366155    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/752e574a-9cd1-40a1-a65e-7b2aabd4434e-tmp\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.366444 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.366190    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-multus-socket-dir-parent\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.366444 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.366223    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50192cd-cb7b-4b3e-978f-661093207b81-tmp-dir\") pod \"node-resolver-q8pg5\" (UID: \"e50192cd-cb7b-4b3e-978f-661093207b81\") " pod="openshift-dns/node-resolver-q8pg5"
Oct 29 22:42:07.366444 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.366253    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-log-socket\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.366444 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.366285    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c68dp\" (UniqueName: \"kubernetes.io/projected/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-kube-api-access-c68dp\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.366444 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.366386    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-modprobe-d\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.366894 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.366469    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-run\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.366894 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.366476    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-slash\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.366894 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.366652    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-run-ovn\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.366894 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.366798    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-systemd-units\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.367104 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.366895    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-var-lib-openvswitch\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.367104 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.366976    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-cni-bin\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.367214 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367103    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-kubernetes\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.367214 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367152    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-etc-kubernetes\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.367214 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367157    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da8029d1-50de-4612-adce-3a3f582c74d8-host\") pod \"node-ca-n27f7\" (UID: \"da8029d1-50de-4612-adce-3a3f582c74d8\") " pod="openshift-image-registry/node-ca-n27f7"
Oct 29 22:42:07.367214 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367205    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-ovnkube-config\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.367413 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367212    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da8029d1-50de-4612-adce-3a3f582c74d8-host\") pod \"node-ca-n27f7\" (UID: \"da8029d1-50de-4612-adce-3a3f582c74d8\") " pod="openshift-image-registry/node-ca-n27f7"
Oct 29 22:42:07.367413 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367217    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-cnibin\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.367413 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367267    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-multus-conf-dir\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.367413 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367303    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d6fdbfe-0357-4139-a034-28faec681123-multus-daemon-config\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.367413 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367311    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-multus-socket-dir-parent\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.367413 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367331    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-etc-openvswitch\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.367413 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367362    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-env-overrides\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.367413 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367364    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-log-socket\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.367413 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367381    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-cnibin\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.367864 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367435    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-multus-conf-dir\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.367864 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367449    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-run\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.367864 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367476    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-device-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.367864 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367518    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/98c4af2a-2166-4fa0-99d0-662be28b6899-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.367864 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367548    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-kubelet\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.367864 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367551    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-device-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.367864 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367593    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs\") pod \"network-metrics-daemon-prbcb\" (UID: \"d7f074e8-1737-4f5c-99cd-928d9c565c17\") " pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:07.367864 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367599    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-kubelet\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.367864 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367631    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-host\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.367864 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367659    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-etc-selinux\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.367864 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367691    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-hostroot\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.367864 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367698    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-host\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.367864 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367743    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0d1b5f20-523b-4c79-a942-b0dfce8f64f9-iptables-alerter-script\") pod \"iptables-alerter-2q4dp\" (UID: \"0d1b5f20-523b-4c79-a942-b0dfce8f64f9\") " pod="openshift-network-operator/iptables-alerter-2q4dp"
Oct 29 22:42:07.367864 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367764    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-etc-selinux\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.367864 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367785    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98c4af2a-2166-4fa0-99d0-662be28b6899-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.367864 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367823    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-etc-openvswitch\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.367864 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367824    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.368728 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367860    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.368728 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367867    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:07.368728 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367917    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-tuned\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.368728 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367952    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98c4af2a-2166-4fa0-99d0-662be28b6899-cnibin\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.368728 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.367985    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb7k9\" (UniqueName: \"kubernetes.io/projected/98c4af2a-2166-4fa0-99d0-662be28b6899-kube-api-access-gb7k9\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.368728 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.368013    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-ovnkube-script-lib\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.368728 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.368045    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d6fdbfe-0357-4139-a034-28faec681123-cni-binary-copy\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.368728 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.368077    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-var-lib-cni-bin\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.368728 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.368159    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/98c4af2a-2166-4fa0-99d0-662be28b6899-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.368728 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.368231    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-run-netns\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.368728 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.368270    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-hostroot\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.368728 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.368305    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-var-lib-cni-bin\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.368728 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.368398    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:07.368728 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.368474    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs podName:d7f074e8-1737-4f5c-99cd-928d9c565c17 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:07.868447292 +0000 UTC m=+3.063269360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs") pod "network-metrics-daemon-prbcb" (UID: "d7f074e8-1737-4f5c-99cd-928d9c565c17") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:07.368728 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.368492    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50192cd-cb7b-4b3e-978f-661093207b81-tmp-dir\") pod \"node-resolver-q8pg5\" (UID: \"e50192cd-cb7b-4b3e-978f-661093207b81\") " pod="openshift-dns/node-resolver-q8pg5"
Oct 29 22:42:07.368728 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.368653    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98c4af2a-2166-4fa0-99d0-662be28b6899-cni-binary-copy\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.368728 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.368697    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ndcg\" (UniqueName: \"kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg\") pod \"network-check-target-lqjl6\" (UID: \"8ba702dc-08af-46e3-8235-61da0c303efd\") " pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:07.369697 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.368735    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fea00aae-5dd9-48ef-b7d4-af08eed1481d-konnectivity-ca\") pod \"konnectivity-agent-5jn2n\" (UID: \"fea00aae-5dd9-48ef-b7d4-af08eed1481d\") " pod="kube-system/konnectivity-agent-5jn2n"
Oct 29 22:42:07.369697 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.368773    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-sysconfig\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.369697 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.368806    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-systemd\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.369697 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.368835    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-var-lib-kubelet\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.369697 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.368886    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.369697 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.368926    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/98c4af2a-2166-4fa0-99d0-662be28b6899-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.369697 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.368954    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fea00aae-5dd9-48ef-b7d4-af08eed1481d-agent-certs\") pod \"konnectivity-agent-5jn2n\" (UID: \"fea00aae-5dd9-48ef-b7d4-af08eed1481d\") " pod="kube-system/konnectivity-agent-5jn2n"
Oct 29 22:42:07.369697 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.368969    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e50192cd-cb7b-4b3e-978f-661093207b81-hosts-file\") pod \"node-resolver-q8pg5\" (UID: \"e50192cd-cb7b-4b3e-978f-661093207b81\") " pod="openshift-dns/node-resolver-q8pg5"
Oct 29 22:42:07.369697 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.369006    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da8029d1-50de-4612-adce-3a3f582c74d8-serviceca\") pod \"node-ca-n27f7\" (UID: \"da8029d1-50de-4612-adce-3a3f582c74d8\") " pod="openshift-image-registry/node-ca-n27f7"
Oct 29 22:42:07.369697 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.369032    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-system-cni-dir\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.369697 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.369048    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-systemd\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.369697 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.369064    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-run-k8s-cni-cncf-io\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.369697 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.369178    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-run-k8s-cni-cncf-io\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.369697 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.369260    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-var-lib-kubelet\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.369697 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.369325    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.369697 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.369511    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:07.369697 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.369579    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret podName:6edb8e5f-e185-48be-8fed-5ac6899134fa nodeName:}" failed. No retries permitted until 2026-10-29 22:42:07.869560922 +0000 UTC m=+3.064383001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret") pod "global-pull-secret-syncer-qmfpg" (UID: "6edb8e5f-e185-48be-8fed-5ac6899134fa") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:07.370583 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.369600    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98c4af2a-2166-4fa0-99d0-662be28b6899-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.370583 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.369685    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-sysconfig\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.370583 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.369727    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0d1b5f20-523b-4c79-a942-b0dfce8f64f9-iptables-alerter-script\") pod \"iptables-alerter-2q4dp\" (UID: \"0d1b5f20-523b-4c79-a942-b0dfce8f64f9\") " pod="openshift-network-operator/iptables-alerter-2q4dp"
Oct 29 22:42:07.370583 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.369787    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e50192cd-cb7b-4b3e-978f-661093207b81-hosts-file\") pod \"node-resolver-q8pg5\" (UID: \"e50192cd-cb7b-4b3e-978f-661093207b81\") " pod="openshift-dns/node-resolver-q8pg5"
Oct 29 22:42:07.370583 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.370007    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-ovnkube-script-lib\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.370583 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.370073    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-sys-fs\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.370583 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.370179    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98c4af2a-2166-4fa0-99d0-662be28b6899-cni-binary-copy\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.370583 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.370283    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-system-cni-dir\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.370583 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.370285    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da8029d1-50de-4612-adce-3a3f582c74d8-serviceca\") pod \"node-ca-n27f7\" (UID: \"da8029d1-50de-4612-adce-3a3f582c74d8\") " pod="openshift-image-registry/node-ca-n27f7"
Oct 29 22:42:07.370583 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.370307    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d6fdbfe-0357-4139-a034-28faec681123-multus-daemon-config\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.370583 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.370326    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98c4af2a-2166-4fa0-99d0-662be28b6899-cnibin\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.370583 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.370493    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d6fdbfe-0357-4139-a034-28faec681123-cni-binary-copy\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.371211 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.370992    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fea00aae-5dd9-48ef-b7d4-af08eed1481d-konnectivity-ca\") pod \"konnectivity-agent-5jn2n\" (UID: \"fea00aae-5dd9-48ef-b7d4-af08eed1481d\") " pod="kube-system/konnectivity-agent-5jn2n"
Oct 29 22:42:07.371211 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.371119    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/98c4af2a-2166-4fa0-99d0-662be28b6899-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.371622 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.371599    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/752e574a-9cd1-40a1-a65e-7b2aabd4434e-tmp\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.374457 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.374432    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-tuned\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.379473 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.379453    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-ovn-node-metrics-cert\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.403725 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.403701    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6ncm\" (UniqueName: \"kubernetes.io/projected/e50192cd-cb7b-4b3e-978f-661093207b81-kube-api-access-l6ncm\") pod \"node-resolver-q8pg5\" (UID: \"e50192cd-cb7b-4b3e-978f-661093207b81\") " pod="openshift-dns/node-resolver-q8pg5"
Oct 29 22:42:07.406825 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.406746    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6q45\" (UniqueName: \"kubernetes.io/projected/765947ed-ccf0-4296-aac8-eace5ce6de20-kube-api-access-p6q45\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.408297 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.407972    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:42:07.408297 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.407998    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:42:07.408297 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.408015    2575 projected.go:196] Error preparing data for projected volume kube-api-access-5ndcg for pod openshift-network-diagnostics/network-check-target-lqjl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:07.408297 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.408092    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg podName:8ba702dc-08af-46e3-8235-61da0c303efd nodeName:}" failed. No retries permitted until 2026-10-29 22:42:07.908074658 +0000 UTC m=+3.102896728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5ndcg" (UniqueName: "kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg") pod "network-check-target-lqjl6" (UID: "8ba702dc-08af-46e3-8235-61da0c303efd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:07.409376 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.409345    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srpcs\" (UniqueName: \"kubernetes.io/projected/7d6fdbfe-0357-4139-a034-28faec681123-kube-api-access-srpcs\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.411003 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.410954    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb7k9\" (UniqueName: \"kubernetes.io/projected/98c4af2a-2166-4fa0-99d0-662be28b6899-kube-api-access-gb7k9\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.412121 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.412105    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qckd7\" (UniqueName: \"kubernetes.io/projected/752e574a-9cd1-40a1-a65e-7b2aabd4434e-kube-api-access-qckd7\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.412245 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.412209    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l295\" (UniqueName: \"kubernetes.io/projected/0d1b5f20-523b-4c79-a942-b0dfce8f64f9-kube-api-access-4l295\") pod \"iptables-alerter-2q4dp\" (UID: \"0d1b5f20-523b-4c79-a942-b0dfce8f64f9\") " pod="openshift-network-operator/iptables-alerter-2q4dp"
Oct 29 22:42:07.412353 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.412312    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c68dp\" (UniqueName: \"kubernetes.io/projected/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-kube-api-access-c68dp\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.412729 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.412707    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f8gp\" (UniqueName: \"kubernetes.io/projected/da8029d1-50de-4612-adce-3a3f582c74d8-kube-api-access-9f8gp\") pod \"node-ca-n27f7\" (UID: \"da8029d1-50de-4612-adce-3a3f582c74d8\") " pod="openshift-image-registry/node-ca-n27f7"
Oct 29 22:42:07.413195 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.413171    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmwh\" (UniqueName: \"kubernetes.io/projected/d7f074e8-1737-4f5c-99cd-928d9c565c17-kube-api-access-6hmwh\") pod \"network-metrics-daemon-prbcb\" (UID: \"d7f074e8-1737-4f5c-99cd-928d9c565c17\") " pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:07.509727 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.509695    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n27f7"
Oct 29 22:42:07.514684 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.514257    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:07.524532 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:07.524485    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8029d1_50de_4612_adce_3a3f582c74d8.slice/crio-ee2095a2270c3e367d66fc11cc6adb6430f788db587db7a903662543a24f764b WatchSource:0}: Error finding container ee2095a2270c3e367d66fc11cc6adb6430f788db587db7a903662543a24f764b: Status 404 returned error can't find the container with id ee2095a2270c3e367d66fc11cc6adb6430f788db587db7a903662543a24f764b
Oct 29 22:42:07.525125 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.525096    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:42:07.525334 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:07.525310    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod501b61a4_5a18_4c9f_8b37_0b162c9c2c2b.slice/crio-8d54764a466fb9eb6014266167e8c7115df9fedebe5929c729e64a5fa96dbae3 WatchSource:0}: Error finding container 8d54764a466fb9eb6014266167e8c7115df9fedebe5929c729e64a5fa96dbae3: Status 404 returned error can't find the container with id 8d54764a466fb9eb6014266167e8c7115df9fedebe5929c729e64a5fa96dbae3
Oct 29 22:42:07.525923 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.525891    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8dtlv"
Oct 29 22:42:07.530285 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.530265    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2q4dp"
Oct 29 22:42:07.537465 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:07.537436    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765947ed_ccf0_4296_aac8_eace5ce6de20.slice/crio-e3fb8defa21a2e126f88bfe892784f4fe6d9ab5399e650c548c56991a01c8746 WatchSource:0}: Error finding container e3fb8defa21a2e126f88bfe892784f4fe6d9ab5399e650c548c56991a01c8746: Status 404 returned error can't find the container with id e3fb8defa21a2e126f88bfe892784f4fe6d9ab5399e650c548c56991a01c8746
Oct 29 22:42:07.545062 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:07.545038    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1b5f20_523b_4c79_a942_b0dfce8f64f9.slice/crio-01d7dbcf2939dfd219ce95104599895c03376407a6809ca7484bad8883735898 WatchSource:0}: Error finding container 01d7dbcf2939dfd219ce95104599895c03376407a6809ca7484bad8883735898: Status 404 returned error can't find the container with id 01d7dbcf2939dfd219ce95104599895c03376407a6809ca7484bad8883735898
Oct 29 22:42:07.556691 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.556665    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:42:07.559132 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.559093    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5jn2n"
Oct 29 22:42:07.562528 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.562505    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q8pg5"
Oct 29 22:42:07.570015 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:07.569985    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c4af2a_2166_4fa0_99d0_662be28b6899.slice/crio-644826cb83d272dc432199658b25ed68b6ac98d886cf927f903de2a1447fcb93 WatchSource:0}: Error finding container 644826cb83d272dc432199658b25ed68b6ac98d886cf927f903de2a1447fcb93: Status 404 returned error can't find the container with id 644826cb83d272dc432199658b25ed68b6ac98d886cf927f903de2a1447fcb93
Oct 29 22:42:07.572392 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.572370    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:42:07.574122 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:07.574048    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea00aae_5dd9_48ef_b7d4_af08eed1481d.slice/crio-32ad4b291e006e99e6a5924e04a9322cd4d78b5f2c43c9fd409adbf6a52e0a3d WatchSource:0}: Error finding container 32ad4b291e006e99e6a5924e04a9322cd4d78b5f2c43c9fd409adbf6a52e0a3d: Status 404 returned error can't find the container with id 32ad4b291e006e99e6a5924e04a9322cd4d78b5f2c43c9fd409adbf6a52e0a3d
Oct 29 22:42:07.578121 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:07.578075    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50192cd_cb7b_4b3e_978f_661093207b81.slice/crio-e435908e8c3b11d7a0a396cdbcc0b995c24c9229caf0774ce203615e976a7209 WatchSource:0}: Error finding container e435908e8c3b11d7a0a396cdbcc0b995c24c9229caf0774ce203615e976a7209: Status 404 returned error can't find the container with id e435908e8c3b11d7a0a396cdbcc0b995c24c9229caf0774ce203615e976a7209
Oct 29 22:42:07.584009 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:07.583975    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752e574a_9cd1_40a1_a65e_7b2aabd4434e.slice/crio-48ad0d731de03f49c4948f0b28ef95d7cdd9d0258228a48de0a7816f7978642f WatchSource:0}: Error finding container 48ad0d731de03f49c4948f0b28ef95d7cdd9d0258228a48de0a7816f7978642f: Status 404 returned error can't find the container with id 48ad0d731de03f49c4948f0b28ef95d7cdd9d0258228a48de0a7816f7978642f
Oct 29 22:42:07.872808 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.872782    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs\") pod \"network-metrics-daemon-prbcb\" (UID: \"d7f074e8-1737-4f5c-99cd-928d9c565c17\") " pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:07.872931 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.872836    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:07.872996 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.872956    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:07.873031 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.872995    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:07.873031 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.873007    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret podName:6edb8e5f-e185-48be-8fed-5ac6899134fa nodeName:}" failed. No retries permitted until 2026-10-29 22:42:08.872991197 +0000 UTC m=+4.067813273 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret") pod "global-pull-secret-syncer-qmfpg" (UID: "6edb8e5f-e185-48be-8fed-5ac6899134fa") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:07.873102 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.873044    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs podName:d7f074e8-1737-4f5c-99cd-928d9c565c17 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:08.873032129 +0000 UTC m=+4.067854203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs") pod "network-metrics-daemon-prbcb" (UID: "d7f074e8-1737-4f5c-99cd-928d9c565c17") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:07.974280 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:07.974206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ndcg\" (UniqueName: \"kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg\") pod \"network-check-target-lqjl6\" (UID: \"8ba702dc-08af-46e3-8235-61da0c303efd\") " pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:07.974454 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.974408    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:42:07.974454 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.974434    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:42:07.974454 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.974449    2575 projected.go:196] Error preparing data for projected volume kube-api-access-5ndcg for pod openshift-network-diagnostics/network-check-target-lqjl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:07.974709 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:07.974510    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg podName:8ba702dc-08af-46e3-8235-61da0c303efd nodeName:}" failed. No retries permitted until 2026-10-29 22:42:08.97448906 +0000 UTC m=+4.169311121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ndcg" (UniqueName: "kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg") pod "network-check-target-lqjl6" (UID: "8ba702dc-08af-46e3-8235-61da0c303efd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:08.221544 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:08.221468    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:37:07 +0000 UTC" deadline="2028-04-30 08:19:07.022654008 +0000 UTC"
Oct 29 22:42:08.221544 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:08.221541    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13161h36m58.801116304s"
Oct 29 22:42:08.288175 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:08.288133    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal" event={"ID":"a47eb3de0ec2692f9f0b4d5a8aa019c1","Type":"ContainerStarted","Data":"2b28a001751eccc381bca7e345083a9d4579d7a556e6d367ece8a55387f4a5c3"}
Oct 29 22:42:08.293715 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:08.293561    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4bgs7" event={"ID":"752e574a-9cd1-40a1-a65e-7b2aabd4434e","Type":"ContainerStarted","Data":"48ad0d731de03f49c4948f0b28ef95d7cdd9d0258228a48de0a7816f7978642f"}
Oct 29 22:42:08.306081 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:08.305916    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlsbk" event={"ID":"98c4af2a-2166-4fa0-99d0-662be28b6899","Type":"ContainerStarted","Data":"644826cb83d272dc432199658b25ed68b6ac98d886cf927f903de2a1447fcb93"}
Oct 29 22:42:08.315113 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:08.315017    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2q4dp" event={"ID":"0d1b5f20-523b-4c79-a942-b0dfce8f64f9","Type":"ContainerStarted","Data":"01d7dbcf2939dfd219ce95104599895c03376407a6809ca7484bad8883735898"}
Oct 29 22:42:08.318157 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:08.318132    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8dtlv" event={"ID":"7d6fdbfe-0357-4139-a034-28faec681123","Type":"ContainerStarted","Data":"98def0669601be1afa30b68afafc9f9bb6e173de5259bb5fd882e3667b492f09"}
Oct 29 22:42:08.325286 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:08.325181    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d" event={"ID":"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b","Type":"ContainerStarted","Data":"8d54764a466fb9eb6014266167e8c7115df9fedebe5929c729e64a5fa96dbae3"}
Oct 29 22:42:08.342262 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:08.342153    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q8pg5" event={"ID":"e50192cd-cb7b-4b3e-978f-661093207b81","Type":"ContainerStarted","Data":"e435908e8c3b11d7a0a396cdbcc0b995c24c9229caf0774ce203615e976a7209"}
Oct 29 22:42:08.351649 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:08.351574    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5jn2n" event={"ID":"fea00aae-5dd9-48ef-b7d4-af08eed1481d","Type":"ContainerStarted","Data":"32ad4b291e006e99e6a5924e04a9322cd4d78b5f2c43c9fd409adbf6a52e0a3d"}
Oct 29 22:42:08.358754 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:08.358684    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666" event={"ID":"765947ed-ccf0-4296-aac8-eace5ce6de20","Type":"ContainerStarted","Data":"e3fb8defa21a2e126f88bfe892784f4fe6d9ab5399e650c548c56991a01c8746"}
Oct 29 22:42:08.362886 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:08.362806    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n27f7" event={"ID":"da8029d1-50de-4612-adce-3a3f582c74d8","Type":"ContainerStarted","Data":"ee2095a2270c3e367d66fc11cc6adb6430f788db587db7a903662543a24f764b"}
Oct 29 22:42:08.883081 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:08.882834    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs\") pod \"network-metrics-daemon-prbcb\" (UID: \"d7f074e8-1737-4f5c-99cd-928d9c565c17\") " pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:08.883265 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:08.883116    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:08.883265 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:08.883162    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:08.883265 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:08.883240    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs podName:d7f074e8-1737-4f5c-99cd-928d9c565c17 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:10.88321725 +0000 UTC m=+6.078039312 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs") pod "network-metrics-daemon-prbcb" (UID: "d7f074e8-1737-4f5c-99cd-928d9c565c17") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:08.883440 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:08.883268    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:08.883440 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:08.883322    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret podName:6edb8e5f-e185-48be-8fed-5ac6899134fa nodeName:}" failed. No retries permitted until 2026-10-29 22:42:10.883304921 +0000 UTC m=+6.078126993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret") pod "global-pull-secret-syncer-qmfpg" (UID: "6edb8e5f-e185-48be-8fed-5ac6899134fa") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:08.983895 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:08.983846    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ndcg\" (UniqueName: \"kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg\") pod \"network-check-target-lqjl6\" (UID: \"8ba702dc-08af-46e3-8235-61da0c303efd\") " pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:08.984083 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:08.984062    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:42:08.984083 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:08.984081    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:42:08.984201 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:08.984094    2575 projected.go:196] Error preparing data for projected volume kube-api-access-5ndcg for pod openshift-network-diagnostics/network-check-target-lqjl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:08.984201 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:08.984150    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg podName:8ba702dc-08af-46e3-8235-61da0c303efd nodeName:}" failed. No retries permitted until 2026-10-29 22:42:10.984130463 +0000 UTC m=+6.178952527 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ndcg" (UniqueName: "kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg") pod "network-check-target-lqjl6" (UID: "8ba702dc-08af-46e3-8235-61da0c303efd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:09.222539 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:09.222373    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:37:07 +0000 UTC" deadline="2028-06-17 20:11:04.079801051 +0000 UTC"
Oct 29 22:42:09.222539 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:09.222426    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14325h28m54.857380448s"
Oct 29 22:42:09.266219 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:09.266018    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:09.266939 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:09.266528    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:09.266939 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:09.266627    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:09.266939 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:09.266763    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:09.266939 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:09.266855    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:09.267238 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:09.266942    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:09.385801 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:09.385753    2575 generic.go:356] "Generic (PLEG): container finished" podID="404a57dfae016c255d65ef53a28af2de" containerID="15487a37e5318be4b7de51ac5bec7c3419861439da83418fe53d4158ec0490d3" exitCode=0
Oct 29 22:42:09.385989 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:09.385867    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal" event={"ID":"404a57dfae016c255d65ef53a28af2de","Type":"ContainerDied","Data":"15487a37e5318be4b7de51ac5bec7c3419861439da83418fe53d4158ec0490d3"}
Oct 29 22:42:09.434263 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:09.434199    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal" podStartSLOduration=3.434180682 podStartE2EDuration="3.434180682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:08.336684728 +0000 UTC m=+3.531506809" watchObservedRunningTime="2026-10-29 22:42:09.434180682 +0000 UTC m=+4.629002761"
Oct 29 22:42:10.413336 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:10.413254    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal" event={"ID":"404a57dfae016c255d65ef53a28af2de","Type":"ContainerStarted","Data":"b776ecba16dec7e42d9d78a3c20459dcac252ca53033112d8df7914f960780cd"}
Oct 29 22:42:10.462515 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:10.462444    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal" podStartSLOduration=4.462424656 podStartE2EDuration="4.462424656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:10.461943877 +0000 UTC m=+5.656765956" watchObservedRunningTime="2026-10-29 22:42:10.462424656 +0000 UTC m=+5.657246739"
Oct 29 22:42:10.915231 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:10.915190    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs\") pod \"network-metrics-daemon-prbcb\" (UID: \"d7f074e8-1737-4f5c-99cd-928d9c565c17\") " pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:10.915446 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:10.915249    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:10.915446 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:10.915413    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:10.915570 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:10.915486    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret podName:6edb8e5f-e185-48be-8fed-5ac6899134fa nodeName:}" failed. No retries permitted until 2026-10-29 22:42:14.915466775 +0000 UTC m=+10.110288845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret") pod "global-pull-secret-syncer-qmfpg" (UID: "6edb8e5f-e185-48be-8fed-5ac6899134fa") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:10.915938 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:10.915898    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:10.916056 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:10.915954    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs podName:d7f074e8-1737-4f5c-99cd-928d9c565c17 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:14.915937139 +0000 UTC m=+10.110759200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs") pod "network-metrics-daemon-prbcb" (UID: "d7f074e8-1737-4f5c-99cd-928d9c565c17") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:11.017019 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:11.016358    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ndcg\" (UniqueName: \"kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg\") pod \"network-check-target-lqjl6\" (UID: \"8ba702dc-08af-46e3-8235-61da0c303efd\") " pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:11.017019 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:11.016556    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:42:11.017019 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:11.016578    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:42:11.017019 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:11.016591    2575 projected.go:196] Error preparing data for projected volume kube-api-access-5ndcg for pod openshift-network-diagnostics/network-check-target-lqjl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:11.017019 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:11.016652    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg podName:8ba702dc-08af-46e3-8235-61da0c303efd nodeName:}" failed. No retries permitted until 2026-10-29 22:42:15.016632084 +0000 UTC m=+10.211454156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ndcg" (UniqueName: "kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg") pod "network-check-target-lqjl6" (UID: "8ba702dc-08af-46e3-8235-61da0c303efd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:11.266918 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:11.266005    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:11.266918 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:11.266142    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:11.266918 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:11.266569    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:11.266918 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:11.266678    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:11.266918 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:11.266750    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:11.266918 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:11.266822    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:11.424911 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:11.424614    2575 generic.go:356] "Generic (PLEG): container finished" podID="98c4af2a-2166-4fa0-99d0-662be28b6899" containerID="51170e1aac8c0b00d08c7f80bbaa4cadc2bc1ea1b5e8bc8fe97604f55af5bfa7" exitCode=0
Oct 29 22:42:11.426651 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:11.426356    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlsbk" event={"ID":"98c4af2a-2166-4fa0-99d0-662be28b6899","Type":"ContainerDied","Data":"51170e1aac8c0b00d08c7f80bbaa4cadc2bc1ea1b5e8bc8fe97604f55af5bfa7"}
Oct 29 22:42:12.429287 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:12.429244    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5jn2n" event={"ID":"fea00aae-5dd9-48ef-b7d4-af08eed1481d","Type":"ContainerStarted","Data":"9c7fb309d34e42883982c05670b232a6d6b2aad3a1ad20fe53031ceeddc5148a"}
Oct 29 22:42:13.064103 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:13.064030    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5jn2n"
Oct 29 22:42:13.267490 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:13.267460    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:13.267670 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:13.267589    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:13.268049 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:13.268009    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:13.268148 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:13.268122    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:13.268213 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:13.268168    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:13.268269 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:13.268238    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:13.435001 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:13.434957    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n27f7" event={"ID":"da8029d1-50de-4612-adce-3a3f582c74d8","Type":"ContainerStarted","Data":"ef4e2a599f8da618617c814b9e2bc3f5f67ffcf11a60dd9748b0875057b502bb"}
Oct 29 22:42:13.484529 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:13.483507    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5jn2n" podStartSLOduration=4.725410613 podStartE2EDuration="8.483484907s" totalImagesPullingTime="3.758074294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:12.47943446 +0000 UTC m=+7.674256541" watchObservedRunningTime="2026-10-29 22:42:13.483484907 +0000 UTC m=+8.678306987"
Oct 29 22:42:14.438814 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:14.438759    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666" event={"ID":"765947ed-ccf0-4296-aac8-eace5ce6de20","Type":"ContainerStarted","Data":"0b645ccbf1a2eb9f517d49285f09d271db7fb233f2287504dd51e24373d3b4f0"}
Oct 29 22:42:14.949496 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:14.948723    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs\") pod \"network-metrics-daemon-prbcb\" (UID: \"d7f074e8-1737-4f5c-99cd-928d9c565c17\") " pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:14.949496 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:14.948780    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:14.949496 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:14.948962    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:14.949496 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:14.949023    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret podName:6edb8e5f-e185-48be-8fed-5ac6899134fa nodeName:}" failed. No retries permitted until 2026-10-29 22:42:22.94900567 +0000 UTC m=+18.143827731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret") pod "global-pull-secret-syncer-qmfpg" (UID: "6edb8e5f-e185-48be-8fed-5ac6899134fa") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:14.949496 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:14.949411    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:14.949496 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:14.949459    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs podName:d7f074e8-1737-4f5c-99cd-928d9c565c17 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:22.949444528 +0000 UTC m=+18.144266596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs") pod "network-metrics-daemon-prbcb" (UID: "d7f074e8-1737-4f5c-99cd-928d9c565c17") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:15.049827 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:15.049175    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ndcg\" (UniqueName: \"kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg\") pod \"network-check-target-lqjl6\" (UID: \"8ba702dc-08af-46e3-8235-61da0c303efd\") " pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:15.049827 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:15.049354    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:42:15.049827 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:15.049376    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:42:15.049827 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:15.049389    2575 projected.go:196] Error preparing data for projected volume kube-api-access-5ndcg for pod openshift-network-diagnostics/network-check-target-lqjl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:15.049827 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:15.049453    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg podName:8ba702dc-08af-46e3-8235-61da0c303efd nodeName:}" failed. No retries permitted until 2026-10-29 22:42:23.049432725 +0000 UTC m=+18.244254798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ndcg" (UniqueName: "kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg") pod "network-check-target-lqjl6" (UID: "8ba702dc-08af-46e3-8235-61da0c303efd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:15.266202 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:15.266162    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:15.266315 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:15.266249    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:15.266527 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:15.266512    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:15.266599 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:15.266583    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:15.266706 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:15.266628    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:15.266706 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:15.266671    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:15.425665 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:15.425642    2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 22:42:15.454200 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:15.454154    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666" event={"ID":"765947ed-ccf0-4296-aac8-eace5ce6de20","Type":"ContainerStarted","Data":"1ef1d742874d6984e3586414a58c1e98eb2e6ad162eb2f228ad48cf42f4a04cc"}
Oct 29 22:42:16.203628 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:16.203508    2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:42:15.425663478Z","UUID":"33eeeda2-812b-4aeb-89e9-278aedc16e8c","Handler":null,"Name":"","Endpoint":""}
Oct 29 22:42:16.205449 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:16.205419    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 22:42:16.205576 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:16.205458    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 22:42:16.459232 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:16.459174    2575 generic.go:356] "Generic (PLEG): container finished" podID="98c4af2a-2166-4fa0-99d0-662be28b6899" containerID="f565f4b0e1303c8fb77d771b1be121537fc2daca4cd71d7457aef55a7cf642b9" exitCode=0
Oct 29 22:42:16.459545 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:16.459468    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlsbk" event={"ID":"98c4af2a-2166-4fa0-99d0-662be28b6899","Type":"ContainerDied","Data":"f565f4b0e1303c8fb77d771b1be121537fc2daca4cd71d7457aef55a7cf642b9"}
Oct 29 22:42:16.462967 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:16.462574    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8dtlv" event={"ID":"7d6fdbfe-0357-4139-a034-28faec681123","Type":"ContainerStarted","Data":"872e5c330ff64310004d9d5e6d6836e1e05b796d9c89648e524ac545bb573651"}
Oct 29 22:42:16.464690 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:16.464668    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q8pg5" event={"ID":"e50192cd-cb7b-4b3e-978f-661093207b81","Type":"ContainerStarted","Data":"855ae6809685a4993ddd00608c9d38eb2ea56385dc19cca7e63442963b728939"}
Oct 29 22:42:16.510795 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:16.510311    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n27f7" podStartSLOduration=6.363787765 podStartE2EDuration="11.510292485s" totalImagesPullingTime="5.14650472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:13.484202463 +0000 UTC m=+8.679024545" watchObservedRunningTime="2026-10-29 22:42:16.510292485 +0000 UTC m=+11.705114563"
Oct 29 22:42:16.563071 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:16.563028    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-8dtlv" podStartSLOduration=3.525914089 podStartE2EDuration="11.563015152s" totalImagesPullingTime="8.037101063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:16.562817473 +0000 UTC m=+11.757639551" watchObservedRunningTime="2026-10-29 22:42:16.563015152 +0000 UTC m=+11.757837227"
Oct 29 22:42:16.608438 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:16.608178    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-q8pg5" podStartSLOduration=3.338423757 podStartE2EDuration="11.608151167s" totalImagesPullingTime="8.26972741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:16.608051486 +0000 UTC m=+11.802873549" watchObservedRunningTime="2026-10-29 22:42:16.608151167 +0000 UTC m=+11.802973248"
Oct 29 22:42:17.243543 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:17.243512    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5jn2n"
Oct 29 22:42:17.244639 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:17.244453    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5jn2n"
Oct 29 22:42:17.265625 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:17.265158    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:17.265625 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:17.265186    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:17.265625 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:17.265220    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:17.265625 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:17.265309    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:17.265625 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:17.265411    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:17.265625 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:17.265485    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:17.470338 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:17.470299    2575 generic.go:356] "Generic (PLEG): container finished" podID="98c4af2a-2166-4fa0-99d0-662be28b6899" containerID="0ff4ee435fdde1c564f59980fc82bc74cfb10c7062ba8f85e349f80974e47a3d" exitCode=0
Oct 29 22:42:17.470781 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:17.470391    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlsbk" event={"ID":"98c4af2a-2166-4fa0-99d0-662be28b6899","Type":"ContainerDied","Data":"0ff4ee435fdde1c564f59980fc82bc74cfb10c7062ba8f85e349f80974e47a3d"}
Oct 29 22:42:17.472674 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:17.472646    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2q4dp" event={"ID":"0d1b5f20-523b-4c79-a942-b0dfce8f64f9","Type":"ContainerStarted","Data":"a9231faa04e8c287a9bc634dabc992cd4f85bf9b9f09126e2dd2d841e6114bcb"}
Oct 29 22:42:17.475893 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:17.475868    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666" event={"ID":"765947ed-ccf0-4296-aac8-eace5ce6de20","Type":"ContainerStarted","Data":"943be1cca8f40fa837b9fcf2db0363632a25b1c006b03a8fcdab58e1be49db60"}
Oct 29 22:42:17.476874 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:17.476850    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5jn2n"
Oct 29 22:42:17.566626 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:17.566582    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2q4dp" podStartSLOduration=4.475465808 podStartE2EDuration="12.566570664s" totalImagesPullingTime="8.091104856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:17.566059349 +0000 UTC m=+12.760881425" watchObservedRunningTime="2026-10-29 22:42:17.566570664 +0000 UTC m=+12.761392740"
Oct 29 22:42:17.657435 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:17.657377    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666" podStartSLOduration=4.181182174 podStartE2EDuration="12.657358315s" totalImagesPullingTime="8.476176141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:17.657222097 +0000 UTC m=+12.852044174" watchObservedRunningTime="2026-10-29 22:42:17.657358315 +0000 UTC m=+12.852180392"
Oct 29 22:42:18.479331 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:18.479242    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4bgs7" event={"ID":"752e574a-9cd1-40a1-a65e-7b2aabd4434e","Type":"ContainerStarted","Data":"d4813a8ba9367499597f3c8b5d44f2f0bf2f79dad4a205010a08586d0b9f3f36"}
Oct 29 22:42:18.482327 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:18.482296    2575 generic.go:356] "Generic (PLEG): container finished" podID="98c4af2a-2166-4fa0-99d0-662be28b6899" containerID="a31ab00e9e3e01baec8d093ecb70fcf6004686ea259d597013513f878b2aaa20" exitCode=0
Oct 29 22:42:18.482458 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:18.482402    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlsbk" event={"ID":"98c4af2a-2166-4fa0-99d0-662be28b6899","Type":"ContainerDied","Data":"a31ab00e9e3e01baec8d093ecb70fcf6004686ea259d597013513f878b2aaa20"}
Oct 29 22:42:18.526629 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:18.526584    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4bgs7" podStartSLOduration=2.185706471 podStartE2EDuration="12.526570721s" totalImagesPullingTime="10.34086425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:18.526029515 +0000 UTC m=+13.720851594" watchObservedRunningTime="2026-10-29 22:42:18.526570721 +0000 UTC m=+13.721392797"
Oct 29 22:42:19.265969 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:19.265937    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:19.265969 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:19.265963    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:19.266184 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:19.265938    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:19.266184 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:19.266075    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:19.266184 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:19.266155    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:19.266282 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:19.266254    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:20.489633 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:20.489603    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlsbk" event={"ID":"98c4af2a-2166-4fa0-99d0-662be28b6899","Type":"ContainerStarted","Data":"75023404ed159ccdd0a7c0601a0c9deac7792df6af04c2752bc84281fa1b6aea"}
Oct 29 22:42:21.265729 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:21.265541    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:21.265924 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:21.265541    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:21.265924 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:21.265843    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:21.266031 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:21.265926    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:21.266031 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:21.265616    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:21.266031 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:21.266012    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:21.495224 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:21.495179    2575 generic.go:356] "Generic (PLEG): container finished" podID="98c4af2a-2166-4fa0-99d0-662be28b6899" containerID="75023404ed159ccdd0a7c0601a0c9deac7792df6af04c2752bc84281fa1b6aea" exitCode=0
Oct 29 22:42:21.495585 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:21.495237    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlsbk" event={"ID":"98c4af2a-2166-4fa0-99d0-662be28b6899","Type":"ContainerDied","Data":"75023404ed159ccdd0a7c0601a0c9deac7792df6af04c2752bc84281fa1b6aea"}
Oct 29 22:42:21.898036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:21.897996    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:22.142726 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:22.142698    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:22.502673 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:22.502648    2575 generic.go:356] "Generic (PLEG): container finished" podID="98c4af2a-2166-4fa0-99d0-662be28b6899" containerID="85c841212b51d71aad4c4cabf9928c8517ed8f52337dfe0e38707bee0da36efe" exitCode=0
Oct 29 22:42:22.503036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:22.502735    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlsbk" event={"ID":"98c4af2a-2166-4fa0-99d0-662be28b6899","Type":"ContainerDied","Data":"85c841212b51d71aad4c4cabf9928c8517ed8f52337dfe0e38707bee0da36efe"}
Oct 29 22:42:22.505070 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:22.505047    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d" event={"ID":"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b","Type":"ContainerStarted","Data":"06c6959c20924d636e099570dd245e4cc95260d7aea960ce5c018f0f5edd8f13"}
Oct 29 22:42:22.505171 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:22.505103    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d" event={"ID":"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b","Type":"ContainerStarted","Data":"6662ff776f3028614e3b88f350ead9690bc26b1dbf16064818d224d418991412"}
Oct 29 22:42:22.505171 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:22.505133    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d" event={"ID":"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b","Type":"ContainerStarted","Data":"a3df8e5d7fb4a5398dc154abaff8a9109a91b647760470e326dc91ba1eb145c4"}
Oct 29 22:42:23.012522 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:23.012487    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs\") pod \"network-metrics-daemon-prbcb\" (UID: \"d7f074e8-1737-4f5c-99cd-928d9c565c17\") " pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:23.012522 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:23.012520    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:23.012723 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:23.012620    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:23.012723 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:23.012621    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:23.012723 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:23.012679    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs podName:d7f074e8-1737-4f5c-99cd-928d9c565c17 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:39.012664433 +0000 UTC m=+34.207486492 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs") pod "network-metrics-daemon-prbcb" (UID: "d7f074e8-1737-4f5c-99cd-928d9c565c17") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:23.012723 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:23.012693    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret podName:6edb8e5f-e185-48be-8fed-5ac6899134fa nodeName:}" failed. No retries permitted until 2026-10-29 22:42:39.012686842 +0000 UTC m=+34.207508896 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret") pod "global-pull-secret-syncer-qmfpg" (UID: "6edb8e5f-e185-48be-8fed-5ac6899134fa") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:23.113111 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:23.113075    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ndcg\" (UniqueName: \"kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg\") pod \"network-check-target-lqjl6\" (UID: \"8ba702dc-08af-46e3-8235-61da0c303efd\") " pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:23.113236 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:23.113183    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:42:23.113236 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:23.113196    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:42:23.113236 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:23.113205    2575 projected.go:196] Error preparing data for projected volume kube-api-access-5ndcg for pod openshift-network-diagnostics/network-check-target-lqjl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:23.113337 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:23.113247    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg podName:8ba702dc-08af-46e3-8235-61da0c303efd nodeName:}" failed. No retries permitted until 2026-10-29 22:42:39.113234311 +0000 UTC m=+34.308056379 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ndcg" (UniqueName: "kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg") pod "network-check-target-lqjl6" (UID: "8ba702dc-08af-46e3-8235-61da0c303efd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:23.264578 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:23.264499    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:23.264690 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:23.264635    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:23.264690 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:23.264638    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:23.264690 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:23.264660    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:23.264787 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:23.264745    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:23.264873 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:23.264853    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:23.509748 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:23.509705    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d" event={"ID":"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b","Type":"ContainerStarted","Data":"d0b973940d3da5ae5025904e960a5db97519ef056eeb4dc52e71ac96bd5fbfa1"}
Oct 29 22:42:23.510192 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:23.509842    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d" event={"ID":"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b","Type":"ContainerStarted","Data":"efd82fcc745b14773076302ed54460b87f18826441c0d9a0cbb0daaff67b1971"}
Oct 29 22:42:23.510192 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:23.509876    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d" event={"ID":"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b","Type":"ContainerStarted","Data":"2f853b5c6498ef5316006c4c0570b36c113a97640f3eb194addcbb5cc5d18b22"}
Oct 29 22:42:23.512971 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:23.512944    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mlsbk" event={"ID":"98c4af2a-2166-4fa0-99d0-662be28b6899","Type":"ContainerStarted","Data":"f67fb1f754d7a736fb31c2b088d5a90a910890e59d858306c0d3d0ea16a18f2a"}
Oct 29 22:42:23.566911 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:23.566829    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mlsbk" podStartSLOduration=8.147283597 podStartE2EDuration="18.56681762s" totalImagesPullingTime="10.419534023s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:23.566773582 +0000 UTC m=+18.761595658" watchObservedRunningTime="2026-10-29 22:42:23.56681762 +0000 UTC m=+18.761639689"
Oct 29 22:42:23.584781 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:23.584746    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:23.708944 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:23.708915    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:23.831129 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:23.831050    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:23.953335 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:23.953305    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:24.079954 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:24.079923    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:24.191948 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:24.191919    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:24.328760 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:24.328726    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:25.265875 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:25.265844    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:25.266283 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:25.265879    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:25.266283 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:25.265956    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:25.266283 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:25.265958    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:25.266283 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:25.266031    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:25.266283 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:25.266123    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:25.271766 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:25.271743    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:25.407704 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:25.407664    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:25.520272 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:25.520202    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d" event={"ID":"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b","Type":"ContainerStarted","Data":"0add77836bd40579da5880abe0032455332a893def8932f89df90f29b82aac5f"}
Oct 29 22:42:27.265580 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:27.265545    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:27.265580 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:27.265562    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:27.266455 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:27.265617    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:27.266455 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:27.265697    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:27.266455 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:27.265752    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:27.266455 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:27.265850    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:27.434886 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:27.434843    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:27.527182 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:27.527113    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d" event={"ID":"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b","Type":"ContainerStarted","Data":"8c21922eff4aad8f58f6ef5f4b0e6256c70e5b3d8a62e40ad74330f9e17bec9d"}
Oct 29 22:42:27.527396 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:27.527372    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:27.542487 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:27.542459    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:27.610724 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:27.610694    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:27.722837 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:27.722645    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d" podStartSLOduration=7.913064642 podStartE2EDuration="22.722633689s" totalImagesPullingTime="14.809569047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:27.640696943 +0000 UTC m=+22.835519019" watchObservedRunningTime="2026-10-29 22:42:27.722633689 +0000 UTC m=+22.917455764"
Oct 29 22:42:27.815305 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:27.815231    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:27.976609 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:27.976572    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:28.092363 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:28.092293    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:28.217294 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:28.217262    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:28.529714 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:28.529685    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:42:28.530263 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:28.530242    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:28.546673 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:28.546649    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:28.767035 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:28.766995    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:28.907112 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:28.907075    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:28.927555 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:28.927530    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qmfpg"]
Oct 29 22:42:28.927682 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:28.927661    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:28.927759 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:28.927740    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:28.934110 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:28.934089    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-prbcb"]
Oct 29 22:42:28.934229 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:28.934181    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:28.934305 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:28.934285    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:28.934719 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:28.934700    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lqjl6"]
Oct 29 22:42:28.934822 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:28.934793    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:28.934914 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:28.934898    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:29.255310 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:29.255220    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:29.531809 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:29.531726    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:42:29.533690 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:29.533656    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:29.663128 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:29.663095    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:30.088511 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:30.088474    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:30.212267 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:30.212224    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:30.265006 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:30.264981    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:30.265134 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:30.265085    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:30.336871 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:30.336836    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:30.459592 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:30.459559    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:30.534106 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:30.534081    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:42:30.772454 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:30.772366    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:30.953374 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:30.953337    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:31.093439 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:31.093374    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:31.265048 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:31.265016    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:31.265048 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:31.265039    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:31.265285 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:31.265128    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:31.265285 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:31.265256    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:31.617721 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:31.617687    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:31.990151 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:31.990057    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:32.134040 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:32.134011    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:32.245915 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:32.245836    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:32.264749 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:32.264725    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:32.264889 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:32.264834    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:32.370832 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:32.370791    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:32.768080 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:32.768044    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:32.773754 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:32.773732    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:32.773932 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:32.773915    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:42:32.784659 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:32.784638    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:42:33.265223 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:33.265190    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:33.265390 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:33.265200    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:33.265390 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:33.265288    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:33.265390 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:33.265368    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:33.356890 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:33.356854    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:33.590720 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:33.590656    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q8pg5_e50192cd-cb7b-4b3e-978f-661093207b81/dns-node-resolver/0.log"
Oct 29 22:42:34.024681 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:34.024647    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:34.173691 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:34.173659    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:34.193958 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:34.193937    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n27f7_da8029d1-50de-4612-adce-3a3f582c74d8/node-ca/0.log"
Oct 29 22:42:34.265349 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:34.265327    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:34.265448 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:34.265428    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:34.586151 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:34.586055    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:34.765608 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:34.765578    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:35.167345 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:35.167310    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:35.265275 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:35.265241    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:35.265434 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:35.265338    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:35.265434 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:35.265407    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:35.265539 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:35.265521    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:35.296045 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:35.296019    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:35.899440 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:35.899409    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:36.046356 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:36.046308    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:36.172937 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:36.172866    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:36.265144 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:36.265112    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:36.265288 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:36.265221    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:36.295910 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:36.295876    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:36.413265 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:36.413237    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:36.563125 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:36.563045    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:36.686554 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:36.686530    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:36.820816 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:36.820729    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:37.156154 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:37.156122    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:37.265499 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:37.265468    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:37.265909 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:37.265476    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:37.265909 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:37.265565    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:37.265909 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:37.265663    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:37.776332 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:37.776297    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:38.265000 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:38.264969    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:38.265156 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:38.265078    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:38.357467 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:38.357437    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:38.578962 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:38.578888    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:38.978669 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:38.978636    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:39.027365 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:39.027340    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs\") pod \"network-metrics-daemon-prbcb\" (UID: \"d7f074e8-1737-4f5c-99cd-928d9c565c17\") " pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:39.027493 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:39.027369    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:39.027493 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:39.027469    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:39.027579 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:39.027510    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret podName:6edb8e5f-e185-48be-8fed-5ac6899134fa nodeName:}" failed. No retries permitted until 2026-10-29 22:43:11.027496458 +0000 UTC m=+66.222318512 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret") pod "global-pull-secret-syncer-qmfpg" (UID: "6edb8e5f-e185-48be-8fed-5ac6899134fa") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:39.027579 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:39.027470    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:39.027648 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:39.027579    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs podName:d7f074e8-1737-4f5c-99cd-928d9c565c17 nodeName:}" failed. No retries permitted until 2026-10-29 22:43:11.027567293 +0000 UTC m=+66.222389360 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs") pod "network-metrics-daemon-prbcb" (UID: "d7f074e8-1737-4f5c-99cd-928d9c565c17") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:39.127718 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:39.127690    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ndcg\" (UniqueName: \"kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg\") pod \"network-check-target-lqjl6\" (UID: \"8ba702dc-08af-46e3-8235-61da0c303efd\") " pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:39.127876 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:39.127857    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:42:39.127876 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:39.127877    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:42:39.127955 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:39.127886    2575 projected.go:196] Error preparing data for projected volume kube-api-access-5ndcg for pod openshift-network-diagnostics/network-check-target-lqjl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:39.127955 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:39.127939    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg podName:8ba702dc-08af-46e3-8235-61da0c303efd nodeName:}" failed. No retries permitted until 2026-10-29 22:43:11.127921278 +0000 UTC m=+66.322743350 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ndcg" (UniqueName: "kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg") pod "network-check-target-lqjl6" (UID: "8ba702dc-08af-46e3-8235-61da0c303efd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:39.173247 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:39.173208    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:39.264721 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:39.264694    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:39.264896 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:39.264836    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:39.264896 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:39.264874    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:39.265007 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:39.264986    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:39.757176 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:39.757139    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:40.170172 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:40.170141    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:40.265233 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:40.265207    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:40.265356 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:40.265301    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:40.356966 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:40.356927    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:40.526374 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:40.526293    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:40.645137 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:40.645110    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:41.220606 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:41.220579    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:41.264663 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:41.264639    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:41.264794 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:41.264719    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:41.264880 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:41.264835    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:41.264934 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:41.264917    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:41.345844 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:41.345788    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:41.472615 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:41.472543    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:42.265310 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:42.265278    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:42.265750 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:42.265411    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:43.264960 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:43.264926    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:43.265245 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:43.264932    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:43.265245 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:43.265047    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:43.265245 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:43.265100    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:44.265379 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:44.265348    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:44.265750 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:44.265444    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:44.508931 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:44.508892    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:44.627108 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:44.627075    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:44.887982 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:44.887899    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:45.050576 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:45.050541    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:45.265297 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:45.265221    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:45.265440 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:45.265364    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:45.265440 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:45.265396    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:45.265869 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:45.265529    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:45.272649 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:45.272628    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:45.389101 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:45.389070    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:46.264907 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:46.264878    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:46.265078 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:46.264982    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:47.264934 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:47.264902    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:47.264934 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:47.264922    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:47.265332 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:47.265019    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqjl6" podUID="8ba702dc-08af-46e3-8235-61da0c303efd"
Oct 29 22:42:47.265332 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:47.265117    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qmfpg" podUID="6edb8e5f-e185-48be-8fed-5ac6899134fa"
Oct 29 22:42:48.254716 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:48.254682    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:48.265521 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:48.265494    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:48.265861 ip-10-0-134-127 kubenswrapper[2575]: E1029 22:42:48.265590    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prbcb" podUID="d7f074e8-1737-4f5c-99cd-928d9c565c17"
Oct 29 22:42:48.462151 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:48.462117    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:48.626454 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:48.626425    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:48.777277 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:48.777243    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:49.022744 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.022658    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:49.160571 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.160543    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeReady"
Oct 29 22:42:49.160696 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.160658    2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 29 22:42:49.229070 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.229043    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dgv4b"]
Oct 29 22:42:49.230685 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.230670    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dgv4b"
Oct 29 22:42:49.235143 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.235123    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dcx8f"]
Oct 29 22:42:49.236647 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.236630    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:42:49.255159 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.255137    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 22:42:49.255267 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.255225    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 22:42:49.255400 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.255378    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 22:42:49.255517 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.255427    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d6hqb\""
Oct 29 22:42:49.255517 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.255430    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 22:42:49.255517 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.255454    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vllbq\""
Oct 29 22:42:49.255517 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.255513    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 22:42:49.256161 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.256146    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dgv4b"]
Oct 29 22:42:49.264904 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.264873    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:42:49.265039 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.264883    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:42:49.270730 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.270703    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dcx8f"]
Oct 29 22:42:49.273781 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.272160    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gpthh"]
Oct 29 22:42:49.274990 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.274968    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:42:49.299190 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.299165    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gpthh"]
Oct 29 22:42:49.305687 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.305663    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75ac2c49-e052-4613-a573-0a8cf1475ad3-data-volume\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:42:49.305783 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.305692    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g26rs\" (UniqueName: \"kubernetes.io/projected/ca99b14b-c183-4966-85c2-b685be64e4cb-kube-api-access-g26rs\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:42:49.305783 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.305729    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca99b14b-c183-4966-85c2-b685be64e4cb-metrics-tls\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:42:49.305892 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.305837    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbs5z\" (UniqueName: \"kubernetes.io/projected/86e95d56-5a28-4cdd-b23d-e56a55de786f-kube-api-access-qbs5z\") pod \"ingress-canary-dgv4b\" (UID: \"86e95d56-5a28-4cdd-b23d-e56a55de786f\") " pod="openshift-ingress-canary/ingress-canary-dgv4b"
Oct 29 22:42:49.305892 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.305884    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75ac2c49-e052-4613-a573-0a8cf1475ad3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:42:49.305959 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.305903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75ac2c49-e052-4613-a573-0a8cf1475ad3-crio-socket\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:42:49.305959 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.305923    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86e95d56-5a28-4cdd-b23d-e56a55de786f-cert\") pod \"ingress-canary-dgv4b\" (UID: \"86e95d56-5a28-4cdd-b23d-e56a55de786f\") " pod="openshift-ingress-canary/ingress-canary-dgv4b"
Oct 29 22:42:49.305959 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.305941    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca99b14b-c183-4966-85c2-b685be64e4cb-tmp-dir\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:42:49.306058 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.305980    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75ac2c49-e052-4613-a573-0a8cf1475ad3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:42:49.306058 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.306007    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q96hr\" (UniqueName: \"kubernetes.io/projected/75ac2c49-e052-4613-a573-0a8cf1475ad3-kube-api-access-q96hr\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:42:49.306223 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.306081    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ca99b14b-c183-4966-85c2-b685be64e4cb-config-volume\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:42:49.356817 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.356778    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 22:42:49.356941 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.356922    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 22:42:49.356995 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.356978    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 22:42:49.357099 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.357085    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 22:42:49.357143 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.357088    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ltz9s\""
Oct 29 22:42:49.357143 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.357106    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nxz4d\""
Oct 29 22:42:49.357143 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.357091    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 22:42:49.357143 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.357095    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mcv8v\""
Oct 29 22:42:49.357271 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.357150    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 22:42:49.357271 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.357201    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 22:42:49.407042 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.407020    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86e95d56-5a28-4cdd-b23d-e56a55de786f-cert\") pod \"ingress-canary-dgv4b\" (UID: \"86e95d56-5a28-4cdd-b23d-e56a55de786f\") " pod="openshift-ingress-canary/ingress-canary-dgv4b"
Oct 29 22:42:49.407135 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.407049    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca99b14b-c183-4966-85c2-b685be64e4cb-tmp-dir\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:42:49.407135 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.407066    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75ac2c49-e052-4613-a573-0a8cf1475ad3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:42:49.407135 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.407097    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q96hr\" (UniqueName: \"kubernetes.io/projected/75ac2c49-e052-4613-a573-0a8cf1475ad3-kube-api-access-q96hr\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:42:49.407284 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.407154    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ca99b14b-c183-4966-85c2-b685be64e4cb-config-volume\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:42:49.407284 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.407175    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75ac2c49-e052-4613-a573-0a8cf1475ad3-data-volume\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:42:49.407284 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.407203    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g26rs\" (UniqueName: \"kubernetes.io/projected/ca99b14b-c183-4966-85c2-b685be64e4cb-kube-api-access-g26rs\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:42:49.407284 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.407237    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca99b14b-c183-4966-85c2-b685be64e4cb-metrics-tls\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:42:49.407284 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.407262    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbs5z\" (UniqueName: \"kubernetes.io/projected/86e95d56-5a28-4cdd-b23d-e56a55de786f-kube-api-access-qbs5z\") pod \"ingress-canary-dgv4b\" (UID: \"86e95d56-5a28-4cdd-b23d-e56a55de786f\") " pod="openshift-ingress-canary/ingress-canary-dgv4b"
Oct 29 22:42:49.407514 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.407296    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75ac2c49-e052-4613-a573-0a8cf1475ad3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:42:49.407514 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.407322    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75ac2c49-e052-4613-a573-0a8cf1475ad3-crio-socket\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:42:49.407514 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.407465    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca99b14b-c183-4966-85c2-b685be64e4cb-tmp-dir\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:42:49.407514 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.407494    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75ac2c49-e052-4613-a573-0a8cf1475ad3-crio-socket\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:42:49.407705 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.407658    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75ac2c49-e052-4613-a573-0a8cf1475ad3-data-volume\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:42:49.408176 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.408021    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ca99b14b-c183-4966-85c2-b685be64e4cb-config-volume\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:42:49.408176 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.408070    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75ac2c49-e052-4613-a573-0a8cf1475ad3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:42:49.410513 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.410492    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75ac2c49-e052-4613-a573-0a8cf1475ad3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:42:49.410513 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.410510    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca99b14b-c183-4966-85c2-b685be64e4cb-metrics-tls\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:42:49.410656 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.410635    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86e95d56-5a28-4cdd-b23d-e56a55de786f-cert\") pod \"ingress-canary-dgv4b\" (UID: \"86e95d56-5a28-4cdd-b23d-e56a55de786f\") " pod="openshift-ingress-canary/ingress-canary-dgv4b"
Oct 29 22:42:49.433017 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.432993    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q96hr\" (UniqueName: \"kubernetes.io/projected/75ac2c49-e052-4613-a573-0a8cf1475ad3-kube-api-access-q96hr\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:42:49.433134 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.433115    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbs5z\" (UniqueName: \"kubernetes.io/projected/86e95d56-5a28-4cdd-b23d-e56a55de786f-kube-api-access-qbs5z\") pod \"ingress-canary-dgv4b\" (UID: \"86e95d56-5a28-4cdd-b23d-e56a55de786f\") " pod="openshift-ingress-canary/ingress-canary-dgv4b"
Oct 29 22:42:49.434954 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.434937    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g26rs\" (UniqueName: \"kubernetes.io/projected/ca99b14b-c183-4966-85c2-b685be64e4cb-kube-api-access-g26rs\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:42:49.542125 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.542071    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dgv4b"
Oct 29 22:42:49.548301 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.548283    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:42:49.593074 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.593039    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:42:49.853390 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.853363    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dcx8f"]
Oct 29 22:42:49.856180 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:49.856148    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca99b14b_c183_4966_85c2_b685be64e4cb.slice/crio-b5231174733af6dfccda009ede094b16d8605f1d544c8650e25602e453f823ed WatchSource:0}: Error finding container b5231174733af6dfccda009ede094b16d8605f1d544c8650e25602e453f823ed: Status 404 returned error can't find the container with id b5231174733af6dfccda009ede094b16d8605f1d544c8650e25602e453f823ed
Oct 29 22:42:49.869176 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.867843    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dgv4b"]
Oct 29 22:42:49.871149 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:49.871126    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e95d56_5a28_4cdd_b23d_e56a55de786f.slice/crio-8e5cbc8c2f59c08d0e89de0714b6cf319448bfee1cf83d33c9f7bd6d439627f5 WatchSource:0}: Error finding container 8e5cbc8c2f59c08d0e89de0714b6cf319448bfee1cf83d33c9f7bd6d439627f5: Status 404 returned error can't find the container with id 8e5cbc8c2f59c08d0e89de0714b6cf319448bfee1cf83d33c9f7bd6d439627f5
Oct 29 22:42:49.898901 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:49.898869    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gpthh"]
Oct 29 22:42:49.901257 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:49.901235    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ac2c49_e052_4613_a573_0a8cf1475ad3.slice/crio-11dab8a95c8e0b059d8aa2c664a8555df7364d89df659eeade4c6d8a0c6e7a7e WatchSource:0}: Error finding container 11dab8a95c8e0b059d8aa2c664a8555df7364d89df659eeade4c6d8a0c6e7a7e: Status 404 returned error can't find the container with id 11dab8a95c8e0b059d8aa2c664a8555df7364d89df659eeade4c6d8a0c6e7a7e
Oct 29 22:42:50.265397 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:50.265351    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:42:50.284127 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:50.283970    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kfq4\""
Oct 29 22:42:50.284127 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:50.284091    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 22:42:50.573745 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:50.573716    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dgv4b" event={"ID":"86e95d56-5a28-4cdd-b23d-e56a55de786f","Type":"ContainerStarted","Data":"8e5cbc8c2f59c08d0e89de0714b6cf319448bfee1cf83d33c9f7bd6d439627f5"}
Oct 29 22:42:50.575439 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:50.575414    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gpthh" event={"ID":"75ac2c49-e052-4613-a573-0a8cf1475ad3","Type":"ContainerStarted","Data":"decbcaf3d72cd8ce8067ce184f4c1837a408c8f1c92340664c56839ebc517737"}
Oct 29 22:42:50.575534 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:50.575464    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gpthh" event={"ID":"75ac2c49-e052-4613-a573-0a8cf1475ad3","Type":"ContainerStarted","Data":"11dab8a95c8e0b059d8aa2c664a8555df7364d89df659eeade4c6d8a0c6e7a7e"}
Oct 29 22:42:50.576651 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:50.576621    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dcx8f" event={"ID":"ca99b14b-c183-4966-85c2-b685be64e4cb","Type":"ContainerStarted","Data":"b5231174733af6dfccda009ede094b16d8605f1d544c8650e25602e453f823ed"}
Oct 29 22:42:51.346841 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.346728    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:51.470626 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.470520    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:51.504728 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.504696    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"]
Oct 29 22:42:51.506869 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.506842    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:42:51.525077 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.525044    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38dfb740-4469-4101-a309-4070aeb2dfe3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:42:51.525213 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.525096    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thwgf\" (UniqueName: \"kubernetes.io/projected/38dfb740-4469-4101-a309-4070aeb2dfe3-kube-api-access-thwgf\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:42:51.525213 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.525147    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38dfb740-4469-4101-a309-4070aeb2dfe3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:42:51.525213 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.525186    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/38dfb740-4469-4101-a309-4070aeb2dfe3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:42:51.539918 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.539891    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 22:42:51.539918 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.539903    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 29 22:42:51.540116 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.539981    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 29 22:42:51.540116 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.539997    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 22:42:51.540116 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.540047    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 29 22:42:51.540116 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.540077    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gx7tn\""
Oct 29 22:42:51.556400 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.556369    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"]
Oct 29 22:42:51.583790 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.583761    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gpthh" event={"ID":"75ac2c49-e052-4613-a573-0a8cf1475ad3","Type":"ContainerStarted","Data":"fd8cba3b9290dd1024a0a4b8af4eb9e3d142a51d7415a2257e53887937baeab8"}
Oct 29 22:42:51.587362 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.587335    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:51.626443 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.626413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38dfb740-4469-4101-a309-4070aeb2dfe3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:42:51.626541 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.626458    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwgf\" (UniqueName: \"kubernetes.io/projected/38dfb740-4469-4101-a309-4070aeb2dfe3-kube-api-access-thwgf\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:42:51.626541 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.626505    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38dfb740-4469-4101-a309-4070aeb2dfe3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:42:51.626616 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.626548    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/38dfb740-4469-4101-a309-4070aeb2dfe3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:42:51.628039 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.627991    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38dfb740-4469-4101-a309-4070aeb2dfe3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:42:51.629566 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.629536    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/38dfb740-4469-4101-a309-4070aeb2dfe3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:42:51.629941 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.629920    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38dfb740-4469-4101-a309-4070aeb2dfe3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:42:51.653795 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.653773    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thwgf\" (UniqueName: \"kubernetes.io/projected/38dfb740-4469-4101-a309-4070aeb2dfe3-kube-api-access-thwgf\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:42:51.830831 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.830712    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:42:51.873377 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:51.873350    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:52.136087 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:52.136042    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"]
Oct 29 22:42:52.138960 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:52.138932    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38dfb740_4469_4101_a309_4070aeb2dfe3.slice/crio-01e301c1d0db1e275053aa92117803e0d542ae1aba7e16cce13e6e629981bb46 WatchSource:0}: Error finding container 01e301c1d0db1e275053aa92117803e0d542ae1aba7e16cce13e6e629981bb46: Status 404 returned error can't find the container with id 01e301c1d0db1e275053aa92117803e0d542ae1aba7e16cce13e6e629981bb46
Oct 29 22:42:52.589384 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:52.589343    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dgv4b" event={"ID":"86e95d56-5a28-4cdd-b23d-e56a55de786f","Type":"ContainerStarted","Data":"cb3ac77abfa44a28c1f2c1a8b5c5285db13c486e7a2a46060aa86a102981684f"}
Oct 29 22:42:52.591189 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:52.591122    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr" event={"ID":"38dfb740-4469-4101-a309-4070aeb2dfe3","Type":"ContainerStarted","Data":"01e301c1d0db1e275053aa92117803e0d542ae1aba7e16cce13e6e629981bb46"}
Oct 29 22:42:52.593507 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:52.593482    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gpthh" event={"ID":"75ac2c49-e052-4613-a573-0a8cf1475ad3","Type":"ContainerStarted","Data":"e2b1d26157e024fb417d62f0f8eda0b623bfc085f0cc6104a0a6236e8cc9ec62"}
Oct 29 22:42:52.595502 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:52.595473    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dcx8f" event={"ID":"ca99b14b-c183-4966-85c2-b685be64e4cb","Type":"ContainerStarted","Data":"1c0ea474c32bafd8faacc8ba3fd51080d8e6f9ad2b50a8389c4b55752786d6b0"}
Oct 29 22:42:52.595609 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:52.595520    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dcx8f" event={"ID":"ca99b14b-c183-4966-85c2-b685be64e4cb","Type":"ContainerStarted","Data":"f5bb35a360210589e680d8af31c9f359b8e09f00cc6334853f01708efd7c125b"}
Oct 29 22:42:52.595609 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:52.595597    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:42:52.638494 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:52.638449    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dgv4b" podStartSLOduration=0.999496173 podStartE2EDuration="3.638435097s" totalImagesPullingTime="2.638938924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:52.638025968 +0000 UTC m=+47.832848041" watchObservedRunningTime="2026-10-29 22:42:52.638435097 +0000 UTC m=+47.833257164"
Oct 29 22:42:52.737508 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:52.737459    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-dcx8f" podStartSLOduration=1.988151158 podStartE2EDuration="3.737444048s" totalImagesPullingTime="1.74929289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:52.689755436 +0000 UTC m=+47.884577512" watchObservedRunningTime="2026-10-29 22:42:52.737444048 +0000 UTC m=+47.932266115"
Oct 29 22:42:52.738180 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:52.738142    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gpthh" podStartSLOduration=1.599845776 podStartE2EDuration="3.738132266s" totalImagesPullingTime="2.13828649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:52.73708259 +0000 UTC m=+47.931904666" watchObservedRunningTime="2026-10-29 22:42:52.738132266 +0000 UTC m=+47.932954342"
Oct 29 22:42:53.341435 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.341404    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-56797698b7-xlsbd"]
Oct 29 22:42:53.343426 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.343411    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.364298 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.363664    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 29 22:42:53.364298 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.363704    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 29 22:42:53.364298 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.363733    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 29 22:42:53.364298 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.363930    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 29 22:42:53.364298 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.364174    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 29 22:42:53.377183 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.377164    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 29 22:42:53.377391 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.377373    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 29 22:42:53.392523 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.392505    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dxz6f\""
Oct 29 22:42:53.393944 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.393927    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 29 22:42:53.408160 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.408127    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56797698b7-xlsbd"]
Oct 29 22:42:53.440998 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.440938    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-oauth-serving-cert\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.441097 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.441002    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl249\" (UniqueName: \"kubernetes.io/projected/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-kube-api-access-wl249\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.441097 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.441070    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-config\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.441173 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.441155    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-trusted-ca-bundle\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.441213 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.441188    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-tmp\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.441293 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.441251    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-serving-cert\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.441293 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.441278    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-service-ca\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.441390 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.441298    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-oauth-config\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.542498 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.542464    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-oauth-serving-cert\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.542647 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.542530    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl249\" (UniqueName: \"kubernetes.io/projected/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-kube-api-access-wl249\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.542647 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.542560    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-config\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.542647 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.542583    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-trusted-ca-bundle\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.542647 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.542607    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-tmp\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.542647 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.542639    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-serving-cert\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.542937 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.542667    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-service-ca\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.542937 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.542696    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-oauth-config\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.543199 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.543160    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-tmp\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.543483 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.543456    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-config\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.543564 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.543520    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-oauth-serving-cert\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.543564 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.543534    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-service-ca\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.543749 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.543728    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-trusted-ca-bundle\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.545617 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.545598    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-oauth-config\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.545840 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.545818    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-serving-cert\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.569096 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.569068    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl249\" (UniqueName: \"kubernetes.io/projected/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-kube-api-access-wl249\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.600190 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.600165    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr" event={"ID":"38dfb740-4469-4101-a309-4070aeb2dfe3","Type":"ContainerStarted","Data":"1fba0283c432f69e653a3e357b7e9ca617c25cdd712ae6c7d531f293c06aeb1e"}
Oct 29 22:42:53.600529 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.600215    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr" event={"ID":"38dfb740-4469-4101-a309-4070aeb2dfe3","Type":"ContainerStarted","Data":"6ecef4ac306326e3fafb2bf7f20753d525600d1ab8c10265691454ee253fd4af"}
Oct 29 22:42:53.653258 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.653237    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:42:53.654110 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.654075    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr" podStartSLOduration=1.61986853 podStartE2EDuration="2.654064435s" totalImagesPullingTime="1.034195905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:53.651942348 +0000 UTC m=+48.846764415" watchObservedRunningTime="2026-10-29 22:42:53.654064435 +0000 UTC m=+48.848886508"
Oct 29 22:42:53.956298 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:53.956241    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56797698b7-xlsbd"]
Oct 29 22:42:53.958404 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:53.958378    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8347f7_98bc_4826_a26d_2ad74ec8b27e.slice/crio-973fa4078033110d1cf17d12d7c74370d627c50584c08597c1dd0b031aeac820 WatchSource:0}: Error finding container 973fa4078033110d1cf17d12d7c74370d627c50584c08597c1dd0b031aeac820: Status 404 returned error can't find the container with id 973fa4078033110d1cf17d12d7c74370d627c50584c08597c1dd0b031aeac820
Oct 29 22:42:54.133268 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:54.133238    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:54.606586 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:54.606545    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56797698b7-xlsbd" event={"ID":"9e8347f7-98bc-4826-a26d-2ad74ec8b27e","Type":"ContainerStarted","Data":"973fa4078033110d1cf17d12d7c74370d627c50584c08597c1dd0b031aeac820"}
Oct 29 22:42:55.501518 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:55.501478    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:55.622178 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:55.622141    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:55.751671 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:55.751592    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:55.889433 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:55.889386    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:56.017780 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.017701    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:56.194755 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.194719    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5pzlf"]
Oct 29 22:42:56.199973 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.199941    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:56.200542 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.200525    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.265722 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.265692    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-sys\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.265892 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.265729    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-tls\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.265892 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.265762    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-textfile\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.265969 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.265935    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-root\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.266009 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.265979    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.266090 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.266072    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-959fd\" (UniqueName: \"kubernetes.io/projected/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-kube-api-access-959fd\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.266139 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.266115    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-metrics-client-ca\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.266218 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.266200    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-wtmp\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.266282 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.266262    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.291929 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.291789    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 29 22:42:56.292053 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.291930    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x4fl8\""
Oct 29 22:42:56.292053 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.291972    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 29 22:42:56.293152 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.293124    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 29 22:42:56.366721 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.366698    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-sys\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.366849 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.366731    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-tls\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.366849 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.366761    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-textfile\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.366849 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.366832    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-sys\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.366987 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.366836    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-root\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.366987 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.366880    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-root\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.366987 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.366899    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.366987 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.366943    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-959fd\" (UniqueName: \"kubernetes.io/projected/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-kube-api-access-959fd\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.366987 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.366983    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-metrics-client-ca\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.367195 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.367013    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-wtmp\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.367195 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.367057    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.367195 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.367068    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-textfile\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.367352 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.367199    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-wtmp\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.367603 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.367583    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-metrics-client-ca\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.367724 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.367703    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.369025 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.369003    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.369463 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.369427    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-tls\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.393310 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.393289    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-959fd\" (UniqueName: \"kubernetes.io/projected/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-kube-api-access-959fd\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.407472 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.407438    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:56.514309 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.514281    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:42:56.522899 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:42:56.522872    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44b4b5f_8aa2_4cf2_b72a_16b5002260b6.slice/crio-cbe23aee528bb0703b9f56eef4e9af48a51c00e2cae7c30ebf7b9a769eaa3ca5 WatchSource:0}: Error finding container cbe23aee528bb0703b9f56eef4e9af48a51c00e2cae7c30ebf7b9a769eaa3ca5: Status 404 returned error can't find the container with id cbe23aee528bb0703b9f56eef4e9af48a51c00e2cae7c30ebf7b9a769eaa3ca5
Oct 29 22:42:56.544856 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.544754    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:56.614091 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.614065    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56797698b7-xlsbd" event={"ID":"9e8347f7-98bc-4826-a26d-2ad74ec8b27e","Type":"ContainerStarted","Data":"acec23f1e3c79e2cfcf49ca0b451890c1b403af2ebe102a46cb2dcf9c835c289"}
Oct 29 22:42:56.615359 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.615332    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5pzlf" event={"ID":"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6","Type":"ContainerStarted","Data":"cbe23aee528bb0703b9f56eef4e9af48a51c00e2cae7c30ebf7b9a769eaa3ca5"}
Oct 29 22:42:56.664908 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.664867    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-56797698b7-xlsbd" podStartSLOduration=1.308797402 podStartE2EDuration="3.664854816s" totalImagesPullingTime="2.356057414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:56.664237617 +0000 UTC m=+51.859059693" watchObservedRunningTime="2026-10-29 22:42:56.664854816 +0000 UTC m=+51.859676889"
Oct 29 22:42:56.668251 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.668230    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:56.871669 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.871633    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:56.999305 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:56.999269    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:57.267914 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:57.267788    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:57.388582 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:57.388544    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:57.505963 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:57.505928    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:57.620225 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:57.620195    2575 generic.go:356] "Generic (PLEG): container finished" podID="c44b4b5f-8aa2-4cf2-b72a-16b5002260b6" containerID="116c137eac71033dda759df4da0b1efbd98f12bd958756467d13198fc82d481f" exitCode=0
Oct 29 22:42:57.620398 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:57.620269    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5pzlf" event={"ID":"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6","Type":"ContainerDied","Data":"116c137eac71033dda759df4da0b1efbd98f12bd958756467d13198fc82d481f"}
Oct 29 22:42:57.649553 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:57.649530    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:58.625036 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:58.624997    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5pzlf" event={"ID":"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6","Type":"ContainerStarted","Data":"6cbb5a1cd0615a34aea971fd4ec10cbb1a4bc39e5147cf971f1fcd23b03ea154"}
Oct 29 22:42:58.625395 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:58.625053    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5pzlf" event={"ID":"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6","Type":"ContainerStarted","Data":"6167c9a24f5914bce354e406026ea8fdab6fcb1e9216478a984cb64c4d30ea45"}
Oct 29 22:42:58.702893 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:58.702842    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5pzlf" podStartSLOduration=1.845038084 podStartE2EDuration="2.702828993s" totalImagesPullingTime="790.531783ms" totalInitContainerRuntime="67.259126ms" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:58.687410033 +0000 UTC m=+53.882232109" watchObservedRunningTime="2026-10-29 22:42:58.702828993 +0000 UTC m=+53.897651074"
Oct 29 22:42:59.397852 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:59.397816    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:59.533537 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:59.533503    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:59.650456 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:59.650349    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:59.814562 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:59.814514    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:59.952017 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:42:59.951939    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:00.645585 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:00.645543    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:00.822892 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:00.822863    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:00.995382 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:00.995309    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:01.148211 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:01.148174    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:02.602230 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:02.602189    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:43:02.798008 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:02.797980    2575 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56797698b7-xlsbd"]
Oct 29 22:43:02.837912 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:02.837886    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bff8847db-vg5bs"]
Oct 29 22:43:02.840191 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:02.840176    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:02.877269 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:02.877246    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bff8847db-vg5bs"]
Oct 29 22:43:02.916414 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:02.916387    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a381148-a1d6-404f-94ca-eed5b9481b7a-tmp\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:02.916414 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:02.916415    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-trusted-ca-bundle\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:02.916555 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:02.916497    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-oauth-config\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:02.916555 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:02.916537    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-config\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:02.916617 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:02.916572    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-serving-cert\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:02.916617 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:02.916590    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtrgr\" (UniqueName: \"kubernetes.io/projected/5a381148-a1d6-404f-94ca-eed5b9481b7a-kube-api-access-gtrgr\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:02.916677 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:02.916623    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-service-ca\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:02.916711 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:02.916671    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-oauth-serving-cert\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:03.017772 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.017724    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-oauth-config\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:03.017772 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.017761    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-config\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:03.018040 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.017878    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-serving-cert\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:03.018040 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.017916    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtrgr\" (UniqueName: \"kubernetes.io/projected/5a381148-a1d6-404f-94ca-eed5b9481b7a-kube-api-access-gtrgr\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:03.018040 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.017967    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-service-ca\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:03.018040 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.017990    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-oauth-serving-cert\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:03.018040 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.018036    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a381148-a1d6-404f-94ca-eed5b9481b7a-tmp\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:03.018232 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.018064    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-trusted-ca-bundle\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:03.018598 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.018577    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-config\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:03.018598 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.018584    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a381148-a1d6-404f-94ca-eed5b9481b7a-tmp\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:03.019105 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.019081    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-trusted-ca-bundle\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:03.019194 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.019125    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-service-ca\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:03.019263 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.019245    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-oauth-serving-cert\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:03.020190 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.020168    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-oauth-config\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:03.020308 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.020291    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-serving-cert\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:03.046898 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.046875    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtrgr\" (UniqueName: \"kubernetes.io/projected/5a381148-a1d6-404f-94ca-eed5b9481b7a-kube-api-access-gtrgr\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:03.108703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.108671    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 22:43:03.113423 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.113402    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.131121 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.131065    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tlsz7\""
Oct 29 22:43:03.134996 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.134971    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 29 22:43:03.135979 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.135955    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 29 22:43:03.136131 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.136114    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 29 22:43:03.136179 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.136171    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 29 22:43:03.149943 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.149748    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:03.158538 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.158513    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:03.166255 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.166231    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 29 22:43:03.166357 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.166291    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 29 22:43:03.167530 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.167510    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 22:43:03.219504 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.219476    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-config\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.219622 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.219526    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.219622 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.219553    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-web-config\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.219622 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.219580    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.219622 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.219603    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.219747 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.219646    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40fb2dc7-336c-4483-93e7-68309e345b22-config-out\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.219747 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.219669    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.219747 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.219689    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.219747 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.219719    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.219747 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.219743    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm689\" (UniqueName: \"kubernetes.io/projected/40fb2dc7-336c-4483-93e7-68309e345b22-kube-api-access-mm689\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.219945 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.219780    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.219945 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.219829    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.219945 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.219850    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.219945 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.219873    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.219945 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.219907    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40fb2dc7-336c-4483-93e7-68309e345b22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.219945 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.219936    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.220506 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.219992    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.220506 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.220017    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.220506 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.220040    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.220506 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.220191    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.232475 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.232405    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dk7bsml1jhsna\""
Oct 29 22:43:03.232475 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.232423    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 29 22:43:03.232475 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.232424    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 29 22:43:03.232475 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.232432    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 29 22:43:03.232475 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.232448    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 29 22:43:03.232475 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.232469    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 29 22:43:03.232714 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.232499    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 29 22:43:03.232714 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.232538    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 29 22:43:03.232977 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.232964    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 29 22:43:03.235960 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.235937    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 29 22:43:03.321231 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.321194    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.321410 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.321234    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.321410 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.321370    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.321512 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.321438    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.321512 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.321478    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-config\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.321512 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.321500    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.321790 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.321677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-web-config\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.321790 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.321718    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.321790 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.321746    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.321790 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.321785    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40fb2dc7-336c-4483-93e7-68309e345b22-config-out\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.322080 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.321827    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.322080 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.321852    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.322080 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.321880    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.322080 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.321906    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm689\" (UniqueName: \"kubernetes.io/projected/40fb2dc7-336c-4483-93e7-68309e345b22-kube-api-access-mm689\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.322080 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.321945    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.322080 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.321974    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.322080 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.321996    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.322080 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.322021    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.322080 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.322047    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40fb2dc7-336c-4483-93e7-68309e345b22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.322080 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.322076    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.322570 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.322148    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.322570 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.322171    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.322570 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.322264    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.323126 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.322751    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.323126 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.323044    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.324560 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.323355    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.324560 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.324486    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.325617 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.324698    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-config\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.325617 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.325126    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.325617 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.325486    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.325617 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.325578    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.326428 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.326386    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.326519 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.326491    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.326519 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.326508    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.326600 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.326563    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.326680 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.326663    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40fb2dc7-336c-4483-93e7-68309e345b22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.327282 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.327262    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40fb2dc7-336c-4483-93e7-68309e345b22-config-out\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.327459 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.327439    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-web-config\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.327678 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.327662    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.349494 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.349450    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm689\" (UniqueName: \"kubernetes.io/projected/40fb2dc7-336c-4483-93e7-68309e345b22-kube-api-access-mm689\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.435415 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.435342    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:03.451994 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.451967    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bff8847db-vg5bs"]
Oct 29 22:43:03.454718 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:43:03.454689    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a381148_a1d6_404f_94ca_eed5b9481b7a.slice/crio-b8e96aa62ac1a645d730795eaee1426f93bf779657e28dfae2de6584d4698f47 WatchSource:0}: Error finding container b8e96aa62ac1a645d730795eaee1426f93bf779657e28dfae2de6584d4698f47: Status 404 returned error can't find the container with id b8e96aa62ac1a645d730795eaee1426f93bf779657e28dfae2de6584d4698f47
Oct 29 22:43:03.642718 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.642683    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bff8847db-vg5bs" event={"ID":"5a381148-a1d6-404f-94ca-eed5b9481b7a","Type":"ContainerStarted","Data":"08a02334529f0003ff8281388c584112623150bfc959c4a7ac12df4773c37929"}
Oct 29 22:43:03.643217 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.642729    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bff8847db-vg5bs" event={"ID":"5a381148-a1d6-404f-94ca-eed5b9481b7a","Type":"ContainerStarted","Data":"b8e96aa62ac1a645d730795eaee1426f93bf779657e28dfae2de6584d4698f47"}
Oct 29 22:43:03.653788 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.653755    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:03.693518 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.693431    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6bff8847db-vg5bs" podStartSLOduration=1.6934184060000002 podStartE2EDuration="1.693418406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:03.692515277 +0000 UTC m=+58.887337353" watchObservedRunningTime="2026-10-29 22:43:03.693418406 +0000 UTC m=+58.888240482"
Oct 29 22:43:03.765011 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:03.764986    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 22:43:03.767150 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:43:03.767112    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40fb2dc7_336c_4483_93e7_68309e345b22.slice/crio-4041352b1166bf40c3c43c4d03741fc69b57864ae22033ff0386eca2c885d552 WatchSource:0}: Error finding container 4041352b1166bf40c3c43c4d03741fc69b57864ae22033ff0386eca2c885d552: Status 404 returned error can't find the container with id 4041352b1166bf40c3c43c4d03741fc69b57864ae22033ff0386eca2c885d552
Oct 29 22:43:04.647918 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:04.647890    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40fb2dc7-336c-4483-93e7-68309e345b22","Type":"ContainerStarted","Data":"25def503b46771084aa879d81063461ce12224e245e01a5f16363e7163376790"}
Oct 29 22:43:04.648265 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:04.647953    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40fb2dc7-336c-4483-93e7-68309e345b22","Type":"ContainerStarted","Data":"4041352b1166bf40c3c43c4d03741fc69b57864ae22033ff0386eca2c885d552"}
Oct 29 22:43:05.654420 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:05.654372    2575 generic.go:356] "Generic (PLEG): container finished" podID="40fb2dc7-336c-4483-93e7-68309e345b22" containerID="25def503b46771084aa879d81063461ce12224e245e01a5f16363e7163376790" exitCode=0
Oct 29 22:43:05.654788 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:05.654452    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40fb2dc7-336c-4483-93e7-68309e345b22","Type":"ContainerDied","Data":"25def503b46771084aa879d81063461ce12224e245e01a5f16363e7163376790"}
Oct 29 22:43:06.152144 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:06.152106    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:06.279023 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:06.278931    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:06.539580 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:06.539497    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:06.725147 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:06.725095    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:09.672510 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:09.672473    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40fb2dc7-336c-4483-93e7-68309e345b22","Type":"ContainerStarted","Data":"0e33e4cc7b9ea275421000589c4faeea8368d05f43e8205178a4debc6743faf0"}
Oct 29 22:43:09.672999 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:09.672546    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40fb2dc7-336c-4483-93e7-68309e345b22","Type":"ContainerStarted","Data":"52f35d3e6a76493f253c2191b526960853f1532a5e01b5f2d4315f7e5c338664"}
Oct 29 22:43:10.179338 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.179305    2575 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bff8847db-vg5bs"]
Oct 29 22:43:10.212918 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.212885    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-774f56585c-2x9nc"]
Oct 29 22:43:10.215238 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.215212    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.242138 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.242114    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774f56585c-2x9nc"]
Oct 29 22:43:10.289191 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.289161    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-service-ca\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.289376 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.289198    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-oauth-serving-cert\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.289376 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.289223    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-trusted-ca-bundle\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.289376 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.289290    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d31c1857-282f-4a04-89a9-883b1c2a7c19-tmp\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.289551 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.289417    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-config\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.289551 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.289478    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-serving-cert\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.289551 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.289543    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-oauth-config\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.289700 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.289598    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grckl\" (UniqueName: \"kubernetes.io/projected/d31c1857-282f-4a04-89a9-883b1c2a7c19-kube-api-access-grckl\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.365602 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.365486    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:10.390532 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.390489    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-config\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.390703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.390558    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-serving-cert\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.390703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.390587    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-oauth-config\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.390703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.390613    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grckl\" (UniqueName: \"kubernetes.io/projected/d31c1857-282f-4a04-89a9-883b1c2a7c19-kube-api-access-grckl\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.390703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.390671    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-service-ca\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.390703 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.390703    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-oauth-serving-cert\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.390989 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.390729    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-trusted-ca-bundle\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.390989 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.390746    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d31c1857-282f-4a04-89a9-883b1c2a7c19-tmp\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.391212 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.391190    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d31c1857-282f-4a04-89a9-883b1c2a7c19-tmp\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.391896 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.391871    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-config\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.392035 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.391874    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-service-ca\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.392035 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.391992    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-oauth-serving-cert\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.392149 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.392066    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-trusted-ca-bundle\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.394080 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.394056    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-serving-cert\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.396222 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.396195    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-oauth-config\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.417343 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.417322    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grckl\" (UniqueName: \"kubernetes.io/projected/d31c1857-282f-4a04-89a9-883b1c2a7c19-kube-api-access-grckl\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.490574 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.490548    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:10.528229 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.528202    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:10.679846 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.679815    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40fb2dc7-336c-4483-93e7-68309e345b22","Type":"ContainerStarted","Data":"70891ac9677b77f4959d3a33aad48ac2e18701972eefcc71f0d81dbe72c07ece"}
Oct 29 22:43:10.680321 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.679886    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40fb2dc7-336c-4483-93e7-68309e345b22","Type":"ContainerStarted","Data":"3ad1da5ab33a1157f4a0cfa7a042f8b540dd8f93bfe3e20f79ff938522cd02fe"}
Oct 29 22:43:10.831336 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:10.831265    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774f56585c-2x9nc"]
Oct 29 22:43:10.834300 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:43:10.834273    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31c1857_282f_4a04_89a9_883b1c2a7c19.slice/crio-f5fa9216f177e87c477e1263a558f14e20de1af6f27bba84a2fe048320d3ff15 WatchSource:0}: Error finding container f5fa9216f177e87c477e1263a558f14e20de1af6f27bba84a2fe048320d3ff15: Status 404 returned error can't find the container with id f5fa9216f177e87c477e1263a558f14e20de1af6f27bba84a2fe048320d3ff15
Oct 29 22:43:11.096867 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.096768    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:43:11.096867 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.096850    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs\") pod \"network-metrics-daemon-prbcb\" (UID: \"d7f074e8-1737-4f5c-99cd-928d9c565c17\") " pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:43:11.113525 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.113495    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 22:43:11.113630 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.113537    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 22:43:11.119787 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.119768    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs\") pod \"network-metrics-daemon-prbcb\" (UID: \"d7f074e8-1737-4f5c-99cd-928d9c565c17\") " pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:43:11.119883 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.119769    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:43:11.197508 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.197488    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ndcg\" (UniqueName: \"kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg\") pod \"network-check-target-lqjl6\" (UID: \"8ba702dc-08af-46e3-8235-61da0c303efd\") " pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:43:11.202397 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.202381    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ltz9s\""
Oct 29 22:43:11.206454 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.206440    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:43:11.213767 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.213754    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 22:43:11.234030 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.234012    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 22:43:11.240398 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.240380    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ndcg\" (UniqueName: \"kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg\") pod \"network-check-target-lqjl6\" (UID: \"8ba702dc-08af-46e3-8235-61da0c303efd\") " pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:43:11.294765 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.294738    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kfq4\""
Oct 29 22:43:11.298910 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.298893    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:43:11.497079 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.497038    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mcv8v\""
Oct 29 22:43:11.500166 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.500141    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:43:11.503155 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.503134    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qmfpg"]
Oct 29 22:43:11.508040 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:43:11.508019    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6edb8e5f_e185_48be_8fed_5ac6899134fa.slice/crio-fd54fd7fa5dda74c458b8e324cb76b12e1d9da0704ee2114408db5a55439c272 WatchSource:0}: Error finding container fd54fd7fa5dda74c458b8e324cb76b12e1d9da0704ee2114408db5a55439c272: Status 404 returned error can't find the container with id fd54fd7fa5dda74c458b8e324cb76b12e1d9da0704ee2114408db5a55439c272
Oct 29 22:43:11.613359 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.613330    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-prbcb"]
Oct 29 22:43:11.615617 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:43:11.615589    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f074e8_1737_4f5c_99cd_928d9c565c17.slice/crio-ca49fcd26f642fae0ad4b0e79ce89067828213cedc2ad2266d2e09ac67c2765b WatchSource:0}: Error finding container ca49fcd26f642fae0ad4b0e79ce89067828213cedc2ad2266d2e09ac67c2765b: Status 404 returned error can't find the container with id ca49fcd26f642fae0ad4b0e79ce89067828213cedc2ad2266d2e09ac67c2765b
Oct 29 22:43:11.684357 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.684323    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-prbcb" event={"ID":"d7f074e8-1737-4f5c-99cd-928d9c565c17","Type":"ContainerStarted","Data":"ca49fcd26f642fae0ad4b0e79ce89067828213cedc2ad2266d2e09ac67c2765b"}
Oct 29 22:43:11.685570 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.685543    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qmfpg" event={"ID":"6edb8e5f-e185-48be-8fed-5ac6899134fa","Type":"ContainerStarted","Data":"fd54fd7fa5dda74c458b8e324cb76b12e1d9da0704ee2114408db5a55439c272"}
Oct 29 22:43:11.687071 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.687046    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774f56585c-2x9nc" event={"ID":"d31c1857-282f-4a04-89a9-883b1c2a7c19","Type":"ContainerStarted","Data":"7bf853c9ce736e09b91d80c35913f123d769773021c0b64317798aab342c812f"}
Oct 29 22:43:11.687161 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.687088    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774f56585c-2x9nc" event={"ID":"d31c1857-282f-4a04-89a9-883b1c2a7c19","Type":"ContainerStarted","Data":"f5fa9216f177e87c477e1263a558f14e20de1af6f27bba84a2fe048320d3ff15"}
Oct 29 22:43:11.690437 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.690415    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40fb2dc7-336c-4483-93e7-68309e345b22","Type":"ContainerStarted","Data":"245b1dfb42a4490266da890eb2b1c69db4a7ae8e5532f625782f7e44d3add70d"}
Oct 29 22:43:11.690514 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.690465    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40fb2dc7-336c-4483-93e7-68309e345b22","Type":"ContainerStarted","Data":"90e20b8633e15e309cb95406f40640f55a8a2b48fa76e3496bf6ad2040ddd9bf"}
Oct 29 22:43:11.736139 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.736100    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-774f56585c-2x9nc" podStartSLOduration=1.736086826 podStartE2EDuration="1.736086826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:11.735642421 +0000 UTC m=+66.930464497" watchObservedRunningTime="2026-10-29 22:43:11.736086826 +0000 UTC m=+66.930908899"
Oct 29 22:43:11.801909 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.801855    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.253875777 podStartE2EDuration="8.80184141s" totalImagesPullingTime="5.531703834s" totalInitContainerRuntime="16.261799ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:11.800374001 +0000 UTC m=+66.995196078" watchObservedRunningTime="2026-10-29 22:43:11.80184141 +0000 UTC m=+66.996663486"
Oct 29 22:43:11.807728 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.807658    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lqjl6"]
Oct 29 22:43:11.820853 ip-10-0-134-127 kubenswrapper[2575]: W1029 22:43:11.820827    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba702dc_08af_46e3_8235_61da0c303efd.slice/crio-acc06e41bfb0dde88ef8a71defe049f7f5046423248f71f9feded43539f333b1 WatchSource:0}: Error finding container acc06e41bfb0dde88ef8a71defe049f7f5046423248f71f9feded43539f333b1: Status 404 returned error can't find the container with id acc06e41bfb0dde88ef8a71defe049f7f5046423248f71f9feded43539f333b1
Oct 29 22:43:11.941267 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:11.941220    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:12.085158 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:12.085121    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:12.707375 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:12.706297    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lqjl6" event={"ID":"8ba702dc-08af-46e3-8235-61da0c303efd","Type":"ContainerStarted","Data":"acc06e41bfb0dde88ef8a71defe049f7f5046423248f71f9feded43539f333b1"}
Oct 29 22:43:13.150532 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:13.150497    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:13.221792 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:13.221718    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:13.348452 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:13.348414    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:13.436361 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:13.436331    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:13.713496 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:13.713458    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-prbcb" event={"ID":"d7f074e8-1737-4f5c-99cd-928d9c565c17","Type":"ContainerStarted","Data":"9cc5835949b94dc4b87cdf73e1b37a4bed3bfa012fda947b3db17e294856643a"}
Oct 29 22:43:13.713969 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:13.713527    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-prbcb" event={"ID":"d7f074e8-1737-4f5c-99cd-928d9c565c17","Type":"ContainerStarted","Data":"75c5b813723da4308502434526a8dd05fae8ec7993451b7478fe55eb8d272ede"}
Oct 29 22:43:13.935792 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:13.935609    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:14.083249 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:14.079428    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:14.505973 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:14.505934    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:14.626664 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:14.626586    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:15.723233 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:15.723184    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lqjl6" event={"ID":"8ba702dc-08af-46e3-8235-61da0c303efd","Type":"ContainerStarted","Data":"ec2144fc18023eb31372c4bd8a2264422b3db69a2d9c089ff355b74bf92ebaec"}
Oct 29 22:43:15.723660 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:15.723277    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:43:15.771228 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:15.771174    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-prbcb" podStartSLOduration=69.739749747 podStartE2EDuration="1m10.771157219s" totalImagesPullingTime="1.031407472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:13.763085787 +0000 UTC m=+68.957907863" watchObservedRunningTime="2026-10-29 22:43:15.771157219 +0000 UTC m=+70.965979297"
Oct 29 22:43:15.772253 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:15.772219    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lqjl6" podStartSLOduration=67.493886581 podStartE2EDuration="1m10.772205657s" totalImagesPullingTime="3.278319076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:15.770709919 +0000 UTC m=+70.965532006" watchObservedRunningTime="2026-10-29 22:43:15.772205657 +0000 UTC m=+70.967027735"
Oct 29 22:43:15.997486 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:15.997396    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:16.130796 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:16.130761    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:16.578393 ip-10-0-134-127 kubenswrapper[2575]: I1029 22:43:16.578285    2575 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:43:16.578429 ip-10-0-134-127 systemd[1]: Stopping Kubernetes Kubelet...
Oct 29 22:43:16.599182 ip-10-0-134-127 systemd[1]: kubelet.service: Deactivated successfully.
Oct 29 22:43:16.599346 ip-10-0-134-127 systemd[1]: Stopped Kubernetes Kubelet.
Oct 29 22:43:16.599397 ip-10-0-134-127 systemd[1]: kubelet.service: Consumed 4.073s CPU time.
Oct 29 22:43:16.615472 ip-10-0-134-127 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 22:43:16.689375 ip-10-0-134-127 kubenswrapper[5428]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:43:16.689375 ip-10-0-134-127 kubenswrapper[5428]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 22:43:16.689375 ip-10-0-134-127 kubenswrapper[5428]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:43:16.689375 ip-10-0-134-127 kubenswrapper[5428]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:43:16.691472 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691382    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:43:16.691472 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691472    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691478    5428 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691481    5428 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691485    5428 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691488    5428 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691491    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691493    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691496    5428 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691500    5428 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691503    5428 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691516    5428 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691519    5428 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691522    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691525    5428 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691528    5428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691531    5428 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691534    5428 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691536    5428 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691539    5428 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:43:16.691543 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691548    5428 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691552    5428 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691555    5428 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691558    5428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691561    5428 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691564    5428 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691566    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691569    5428 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691572    5428 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691574    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691577    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691579    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691582    5428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691584    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691587    5428 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691589    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691592    5428 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691595    5428 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691597    5428 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691600    5428 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:43:16.692030 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691603    5428 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691605    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691608    5428 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691611    5428 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691614    5428 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691616    5428 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691619    5428 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691622    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691624    5428 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691627    5428 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691629    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691632    5428 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691634    5428 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691637    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691640    5428 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691644    5428 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691649    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691653    5428 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691655    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:43:16.692541 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691658    5428 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691661    5428 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691663    5428 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691666    5428 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691668    5428 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691671    5428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691674    5428 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691677    5428 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691679    5428 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691682    5428 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691685    5428 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691687    5428 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691690    5428 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691693    5428 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691696    5428 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691699    5428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691701    5428 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691704    5428 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691707    5428 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691711    5428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:43:16.693036 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691714    5428 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691717    5428 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691720    5428 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691722    5428 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691725    5428 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691727    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691730    5428 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691733    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691736    5428 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691738    5428 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691741    5428 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691744    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691746    5428 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691749    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691752    5428 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691755    5428 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691758    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691761    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691763    5428 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:43:16.693526 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691768    5428 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691771    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691774    5428 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691777    5428 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691780    5428 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691782    5428 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691785    5428 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.691788    5428 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692626    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692634    5428 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692638    5428 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692641    5428 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692644    5428 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692647    5428 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692650    5428 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692652    5428 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692655    5428 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692658    5428 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692661    5428 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:43:16.694035 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692664    5428 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692666    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692669    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692672    5428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692675    5428 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692678    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692680    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692683    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692686    5428 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692688    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692691    5428 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692694    5428 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692696    5428 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692699    5428 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692702    5428 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692704    5428 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692707    5428 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692710    5428 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692712    5428 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692715    5428 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:43:16.694514 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692718    5428 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692721    5428 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692723    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692726    5428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692729    5428 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692731    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692735    5428 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692737    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692740    5428 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692742    5428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692745    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692747    5428 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692750    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692753    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692756    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692759    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692762    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692765    5428 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692767    5428 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692770    5428 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:43:16.695043 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692772    5428 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692775    5428 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692778    5428 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692781    5428 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692783    5428 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692786    5428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692789    5428 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692792    5428 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692794    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692797    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692815    5428 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692819    5428 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692823    5428 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692826    5428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692829    5428 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692832    5428 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692836    5428 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692839    5428 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692842    5428 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:43:16.695524 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692845    5428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692847    5428 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692850    5428 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692853    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692856    5428 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692859    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692863    5428 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692867    5428 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692870    5428 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692873    5428 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692876    5428 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692879    5428 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692881    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692884    5428 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692887    5428 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692889    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692892    5428 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692895    5428 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692897    5428 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:43:16.696014 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692903    5428 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692906    5428 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692908    5428 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692911    5428 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692914    5428 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692917    5428 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692920    5428 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692922    5428 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692925    5428 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692928    5428 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692932    5428 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692935    5428 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692937    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692940    5428 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692943    5428 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692945    5428 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.692948    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693453    5428 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693457    5428 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693461    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:43:16.696504 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693463    5428 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693466    5428 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693469    5428 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693472    5428 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693475    5428 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693477    5428 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693480    5428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693483    5428 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693485    5428 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693488    5428 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693491    5428 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693494    5428 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693497    5428 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693500    5428 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693502    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693505    5428 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693507    5428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693510    5428 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693513    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:43:16.697016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693515    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693518    5428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693526    5428 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693529    5428 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693532    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693535    5428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693537    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693541    5428 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693544    5428 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693546    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693549    5428 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693552    5428 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693554    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693557    5428 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693560    5428 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693562    5428 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693565    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693567    5428 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693570    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:43:16.697561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693573    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693575    5428 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693578    5428 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693580    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693583    5428 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693586    5428 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693589    5428 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693592    5428 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693594    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693597    5428 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693600    5428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693602    5428 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693605    5428 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693607    5428 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693610    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693612    5428 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693620    5428 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693623    5428 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693626    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693629    5428 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:43:16.698460 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693632    5428 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693635    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693637    5428 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693640    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693642    5428 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693645    5428 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693647    5428 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693650    5428 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693653    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693655    5428 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693658    5428 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693665    5428 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693667    5428 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693670    5428 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693672    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693676    5428 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693681    5428 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693684    5428 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693688    5428 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:43:16.699329 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693690    5428 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693693    5428 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693696    5428 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693698    5428 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693701    5428 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693703    5428 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693706    5428 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693709    5428 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693711    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693714    5428 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693724    5428 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693728    5428 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693732    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693735    5428 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693738    5428 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693740    5428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693743    5428 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693746    5428 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693748    5428 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:43:16.700152 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693751    5428 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693753    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693757    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693760    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693762    5428 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693766    5428 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.693769    5428 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693874    5428 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693882    5428 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693889    5428 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693893    5428 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693898    5428 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693901    5428 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693906    5428 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693911    5428 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693914    5428 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693917    5428 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693920    5428 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693924    5428 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693927    5428 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693930    5428 flags.go:64] FLAG: --cgroup-root=""
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693933    5428 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693937    5428 flags.go:64] FLAG: --client-ca-file=""
Oct 29 22:43:16.700997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693940    5428 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693943    5428 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693947    5428 flags.go:64] FLAG: --cluster-domain=""
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693950    5428 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693954    5428 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693957    5428 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693960    5428 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693964    5428 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693967    5428 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693971    5428 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693974    5428 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693977    5428 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693980    5428 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693984    5428 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693989    5428 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693992    5428 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693996    5428 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.693999    5428 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694003    5428 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694006    5428 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694009    5428 flags.go:64] FLAG: --enable-server="true"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694012    5428 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694020    5428 flags.go:64] FLAG: --event-burst="100"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694023    5428 flags.go:64] FLAG: --event-qps="50"
Oct 29 22:43:16.701589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694026    5428 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694030    5428 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694033    5428 flags.go:64] FLAG: --eviction-hard=""
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694037    5428 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694040    5428 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694044    5428 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694047    5428 flags.go:64] FLAG: --eviction-soft=""
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694050    5428 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694053    5428 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694056    5428 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694065    5428 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694068    5428 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694071    5428 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694074    5428 flags.go:64] FLAG: --feature-gates=""
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694078    5428 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694082    5428 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694085    5428 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694088    5428 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694091    5428 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694094    5428 flags.go:64] FLAG: --help="false"
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694097    5428 flags.go:64] FLAG: --hostname-override="ip-10-0-134-127.ec2.internal"
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694102    5428 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694106    5428 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 22:43:16.702207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694109    5428 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694113    5428 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694116    5428 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694119    5428 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694122    5428 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694125    5428 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694128    5428 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694131    5428 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694134    5428 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694137    5428 flags.go:64] FLAG: --kube-reserved=""
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694140    5428 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694143    5428 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694147    5428 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694150    5428 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694153    5428 flags.go:64] FLAG: --lock-file=""
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694155    5428 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694158    5428 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694161    5428 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694167    5428 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694170    5428 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694174    5428 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694177    5428 flags.go:64] FLAG: --logging-format="text"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694180    5428 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694184    5428 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 22:43:16.702783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694187    5428 flags.go:64] FLAG: --manifest-url=""
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694190    5428 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694194    5428 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694197    5428 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694201    5428 flags.go:64] FLAG: --max-pods="110"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694204    5428 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694209    5428 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694213    5428 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694216    5428 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694219    5428 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694223    5428 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694225    5428 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694243    5428 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694246    5428 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694250    5428 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694253    5428 flags.go:64] FLAG: --pod-cidr=""
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694256    5428 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694259    5428 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694262    5428 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694265    5428 flags.go:64] FLAG: --port="10250"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694268    5428 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694271    5428 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a8abc33e4f6cca8c"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694274    5428 flags.go:64] FLAG: --qos-reserved=""
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694277    5428 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694280    5428 flags.go:64] FLAG: --register-node="true"
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694283    5428 flags.go:64] FLAG: --register-with-taints=""
Oct 29 22:43:16.703384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694296    5428 flags.go:64] FLAG: --registry-burst="10"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694300    5428 flags.go:64] FLAG: --registry-qps="5"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694303    5428 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694306    5428 flags.go:64] FLAG: --reserved-memory=""
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694311    5428 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694314    5428 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694317    5428 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694320    5428 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694323    5428 flags.go:64] FLAG: --runonce="false"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694326    5428 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694329    5428 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694332    5428 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694336    5428 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694341    5428 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694344    5428 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694347    5428 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694350    5428 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694354    5428 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694357    5428 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694360    5428 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694363    5428 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694366    5428 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694369    5428 flags.go:64] FLAG: --system-cgroups=""
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694372    5428 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694377    5428 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 22:43:16.704167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694380    5428 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694383    5428 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694390    5428 flags.go:64] FLAG: --tls-min-version=""
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694393    5428 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694396    5428 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694399    5428 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694419    5428 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694423    5428 flags.go:64] FLAG: --v="2"
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694428    5428 flags.go:64] FLAG: --version="false"
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694432    5428 flags.go:64] FLAG: --vmodule=""
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694436    5428 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694441    5428 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694580    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694585    5428 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694589    5428 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694593    5428 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694596    5428 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694599    5428 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694602    5428 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694606    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694610    5428 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694613    5428 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:43:16.704825 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694616    5428 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694619    5428 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694621    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694624    5428 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694627    5428 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694629    5428 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694632    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694635    5428 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694637    5428 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694640    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694643    5428 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694645    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694648    5428 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694650    5428 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694653    5428 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694656    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694659    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694661    5428 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694664    5428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694666    5428 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:43:16.705644 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694669    5428 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694672    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694675    5428 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694678    5428 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694681    5428 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694683    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694686    5428 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694689    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694692    5428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694696    5428 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694700    5428 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694703    5428 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694706    5428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694708    5428 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694711    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694714    5428 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694716    5428 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694719    5428 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694722    5428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694724    5428 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:43:16.706309 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694727    5428 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694730    5428 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694733    5428 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694735    5428 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694738    5428 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694741    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694743    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694746    5428 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694748    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694751    5428 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694753    5428 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694756    5428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694758    5428 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694761    5428 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694764    5428 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694767    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694770    5428 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694773    5428 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694775    5428 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:43:16.707079 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694778    5428 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694780    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694785    5428 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694790    5428 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694794    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694797    5428 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694815    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694819    5428 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694823    5428 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694827    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694831    5428 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694835    5428 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694840    5428 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694843    5428 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694845    5428 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694848    5428 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694851    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694853    5428 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694856    5428 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:43:16.707947 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694859    5428 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:43:16.708566 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694862    5428 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:43:16.708566 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694864    5428 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:43:16.708566 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694867    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:43:16.708566 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694870    5428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:43:16.708566 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694873    5428 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:43:16.708566 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694875    5428 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:43:16.708566 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694878    5428 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:43:16.708566 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694881    5428 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:43:16.708566 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694883    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:43:16.708566 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694887    5428 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:43:16.708566 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694890    5428 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:43:16.708566 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694893    5428 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:43:16.708566 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694895    5428 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:43:16.708566 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694898    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:43:16.708566 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694903    5428 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:43:16.708566 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694907    5428 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:43:16.708566 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694910    5428 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:43:16.709162 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.694936    5428 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:43:16.709162 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694970    5428 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:43:16.709162 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694975    5428 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:43:16.709162 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694978    5428 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:43:16.709162 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694981    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:43:16.709162 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694984    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:43:16.709162 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694986    5428 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:43:16.709162 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694989    5428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:43:16.709162 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694992    5428 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:43:16.709162 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694994    5428 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:43:16.709162 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.694997    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:43:16.709162 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695000    5428 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:43:16.709162 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695003    5428 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:43:16.709162 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695006    5428 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:43:16.709162 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695008    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:43:16.709162 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695011    5428 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695013    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695016    5428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695018    5428 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695021    5428 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695023    5428 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695026    5428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695028    5428 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695031    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695034    5428 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695037    5428 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695039    5428 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695042    5428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695045    5428 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695049    5428 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695052    5428 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695056    5428 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695058    5428 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695061    5428 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695064    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:43:16.709571 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695066    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695069    5428 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695071    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695074    5428 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695076    5428 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695079    5428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695081    5428 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695084    5428 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695087    5428 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695090    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695092    5428 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695095    5428 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695097    5428 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695100    5428 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695102    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695105    5428 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695109    5428 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695113    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695116    5428 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:43:16.710100 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695119    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695122    5428 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695125    5428 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695127    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695130    5428 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695133    5428 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695136    5428 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695139    5428 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695142    5428 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695145    5428 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695147    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695150    5428 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695152    5428 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695155    5428 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695158    5428 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695161    5428 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695164    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695166    5428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695169    5428 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:43:16.710565 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695184    5428 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695189    5428 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695192    5428 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695195    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695198    5428 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695201    5428 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695204    5428 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695206    5428 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695209    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695212    5428 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695214    5428 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695217    5428 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695220    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695224    5428 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695227    5428 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695231    5428 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695233    5428 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695237    5428 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695240    5428 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695243    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:43:16.711033 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695247    5428 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695250    5428 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695252    5428 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695255    5428 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695258    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695260    5428 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695263    5428 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695265    5428 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695268    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695270    5428 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695273    5428 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695275    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695278    5428 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.695281    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.699831    5428 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.699849    5428 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.699874    5428 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.699882    5428 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700097    5428 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:43:16.711552 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700123    5428 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700132    5428 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700146    5428 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700152    5428 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700157    5428 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700162    5428 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700167    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700172    5428 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700176    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700181    5428 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700186    5428 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700192    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700197    5428 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700202    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700209    5428 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700220    5428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700225    5428 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700231    5428 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700236    5428 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700242    5428 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:43:16.712092 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700247    5428 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700252    5428 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700257    5428 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700263    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700269    5428 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700275    5428 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700281    5428 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700290    5428 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700295    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700300    5428 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700304    5428 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700309    5428 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700314    5428 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700319    5428 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700324    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700328    5428 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700333    5428 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700338    5428 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700343    5428 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:43:16.712755 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700348    5428 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700357    5428 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700363    5428 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700368    5428 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700372    5428 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700378    5428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700383    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700388    5428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700393    5428 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700397    5428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700402    5428 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700407    5428 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700412    5428 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700422    5428 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700427    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700431    5428 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700436    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700442    5428 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700447    5428 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700452    5428 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:43:16.713560 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700456    5428 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700461    5428 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700466    5428 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700471    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700475    5428 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700480    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700489    5428 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700494    5428 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700499    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700504    5428 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700508    5428 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700513    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700518    5428 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700522    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700527    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700531    5428 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700536    5428 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700541    5428 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700547    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700556    5428 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:43:16.714163 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700561    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700566    5428 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700570    5428 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700575    5428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700580    5428 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700585    5428 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700590    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700595    5428 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700600    5428 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700604    5428 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700609    5428 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700619    5428 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700624    5428 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700629    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700633    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700638    5428 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700643    5428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700648    5428 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700652    5428 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700657    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:43:16.714647 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700661    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:43:16.715138 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700666    5428 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:43:16.715138 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700670    5428 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:43:16.715138 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700675    5428 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:43:16.715138 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700684    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:43:16.715138 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.700689    5428 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:43:16.715138 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.700738    5428 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:43:16.715138 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701495    5428 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:43:16.715138 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701510    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:43:16.715138 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701514    5428 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:43:16.715138 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701517    5428 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:43:16.715138 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701522    5428 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:43:16.715138 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701525    5428 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:43:16.715138 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701528    5428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:43:16.715138 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701531    5428 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:43:16.715138 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701534    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:43:16.715138 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701536    5428 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701540    5428 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701545    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701548    5428 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701551    5428 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701554    5428 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701557    5428 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701560    5428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701562    5428 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701566    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701569    5428 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701571    5428 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701574    5428 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701577    5428 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701580    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701583    5428 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701587    5428 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701590    5428 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701593    5428 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:43:16.715520 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701596    5428 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701599    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701602    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701605    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701608    5428 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701610    5428 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701613    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701616    5428 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701623    5428 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701627    5428 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701629    5428 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701632    5428 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701635    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701637    5428 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701640    5428 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701642    5428 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701645    5428 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701648    5428 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701650    5428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701653    5428 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:43:16.716049 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701655    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701658    5428 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701660    5428 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701663    5428 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701667    5428 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701669    5428 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701672    5428 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701675    5428 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701678    5428 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701680    5428 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701683    5428 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701685    5428 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701688    5428 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701690    5428 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701693    5428 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701695    5428 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701698    5428 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701701    5428 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701703    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701706    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:43:16.716837 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701709    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701711    5428 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701713    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701716    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701719    5428 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701721    5428 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701724    5428 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701727    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701729    5428 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701732    5428 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701735    5428 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701737    5428 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701740    5428 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701742    5428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701745    5428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701748    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701750    5428 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701753    5428 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701756    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:43:16.717688 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701759    5428 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:43:16.718622 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701762    5428 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:43:16.718622 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701764    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:43:16.718622 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701767    5428 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:43:16.718622 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701770    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:43:16.718622 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701773    5428 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:43:16.718622 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701775    5428 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:43:16.718622 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701778    5428 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:43:16.718622 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701781    5428 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:43:16.718622 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701783    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:43:16.718622 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701786    5428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:43:16.718622 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701789    5428 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:43:16.718622 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701791    5428 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:43:16.718622 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701794    5428 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:43:16.718622 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701797    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:43:16.718622 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701818    5428 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:43:16.718622 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701822    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:43:16.718622 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701826    5428 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:43:16.718622 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701829    5428 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:43:16.719561 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.701857    5428 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:43:16.719561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701892    5428 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:43:16.719561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701898    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:43:16.719561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701901    5428 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:43:16.719561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701904    5428 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:43:16.719561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701907    5428 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:43:16.719561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701910    5428 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:43:16.719561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701912    5428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:43:16.719561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701915    5428 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:43:16.719561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701918    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:43:16.719561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701921    5428 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:43:16.719561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701925    5428 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:43:16.719561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701929    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:43:16.719561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701933    5428 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:43:16.719561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701936    5428 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:43:16.719561 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701939    5428 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701942    5428 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701946    5428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701948    5428 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701951    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701954    5428 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701957    5428 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701960    5428 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701962    5428 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701965    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701969    5428 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701972    5428 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701975    5428 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701978    5428 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701981    5428 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701983    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701986    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701988    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701991    5428 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:43:16.720252 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701994    5428 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701997    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.701999    5428 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702002    5428 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702005    5428 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702007    5428 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702012    5428 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702015    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702018    5428 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702020    5428 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702023    5428 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702025    5428 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702028    5428 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702031    5428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702033    5428 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702036    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702038    5428 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702041    5428 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702043    5428 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702046    5428 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:43:16.720957 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702050    5428 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702053    5428 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702056    5428 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702058    5428 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702061    5428 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702064    5428 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702066    5428 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702069    5428 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702072    5428 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702074    5428 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702077    5428 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702080    5428 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702083    5428 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702085    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702088    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702091    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702094    5428 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702097    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702101    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702104    5428 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:43:16.721501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702107    5428 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702110    5428 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702112    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702115    5428 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702118    5428 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702120    5428 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702123    5428 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702125    5428 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702128    5428 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702130    5428 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702133    5428 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702136    5428 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702140    5428 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702143    5428 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702145    5428 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702148    5428 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702151    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702154    5428 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702156    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:43:16.722037 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702159    5428 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:43:16.722627 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702162    5428 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:43:16.722627 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702164    5428 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:43:16.722627 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702167    5428 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:43:16.722627 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702170    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:43:16.722627 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702172    5428 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:43:16.722627 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702175    5428 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:43:16.722627 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702177    5428 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:43:16.722627 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702180    5428 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:43:16.722627 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702183    5428 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:43:16.722627 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702186    5428 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:43:16.722627 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702189    5428 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:43:16.722627 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702192    5428 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:43:16.722627 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:16.702195    5428 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:43:16.722627 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.702253    5428 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 22:43:16.722627 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.703213    5428 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 29 22:43:16.722627 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.703269    5428 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 29 22:43:16.722627 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.703444    5428 server.go:1014] "Starting client certificate rotation"
Oct 29 22:43:16.722627 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.703590    5428 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:43:16.723102 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.703657    5428 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:37:05 +0000 UTC" deadline="2028-07-16 08:49:02.964741674 +0000 UTC"
Oct 29 22:43:16.723102 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.703696    5428 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15010h5m46.261048915s"
Oct 29 22:43:16.723102 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.703901    5428 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:43:16.723102 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.704950    5428 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:43:16.723102 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.706198    5428 log.go:25] "Validated CRI v1 runtime API"
Oct 29 22:43:16.723102 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.708399    5428 log.go:25] "Validated CRI v1 image API"
Oct 29 22:43:16.723102 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.709054    5428 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 22:43:16.723102 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.710923    5428 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 80e69dd1-2d10-4b80-adfc-3e8e869fafd9:/dev/nvme0n1p4 9226ec9e-39c2-487f-b83e-e498690ea092:/dev/nvme0n1p3]
Oct 29 22:43:16.723337 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.710945    5428 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01d7dbcf2939dfd219ce95104599895c03376407a6809ca7484bad8883735898/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01d7dbcf2939dfd219ce95104599895c03376407a6809ca7484bad8883735898/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01e301c1d0db1e275053aa92117803e0d542ae1aba7e16cce13e6e629981bb46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01e301c1d0db1e275053aa92117803e0d542ae1aba7e16cce13e6e629981bb46/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11dab8a95c8e0b059d8aa2c664a8555df7364d89df659eeade4c6d8a0c6e7a7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11dab8a95c8e0b059d8aa2c664a8555df7364d89df659eeade4c6d8a0c6e7a7e/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32ad4b291e006e99e6a5924e04a9322cd4d78b5f2c43c9fd409adbf6a52e0a3d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32ad4b291e006e99e6a5924e04a9322cd4d78b5f2c43c9fd409adbf6a52e0a3d/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4041352b1166bf40c3c43c4d03741fc69b57864ae22033ff0386eca2c885d552/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4041352b1166bf40c3c43c4d03741fc69b57864ae22033ff0386eca2c885d552/userdata/shm major:0 minor:315 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58676504ac4c1ffb5167915e7395604a0c201844b2e26d83f32c3a64cc07a6f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58676504ac4c1ffb5167915e7395604a0c201844b2e26d83f32c3a64cc07a6f2/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/644826cb83d272dc432199658b25ed68b6ac98d886cf927f903de2a1447fcb93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/644826cb83d272dc432199658b25ed68b6ac98d886cf927f903de2a1447fcb93/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78f01703269066ee6b5be698312e3856eee8652f2fbbf9dcf4f145eb25743e8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78f01703269066ee6b5be698312e3856eee8652f2fbbf9dcf4f145eb25743e8a/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d54764a466fb9eb6014266167e8c7115df9fedebe5929c729e64a5fa96dbae3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d54764a466fb9eb6014266167e8c7115df9fedebe5929c729e64a5fa96dbae3/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e5cbc8c2f59c08d0e89de0714b6cf319448bfee1cf83d33c9f7bd6d439627f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e5cbc8c2f59c08d0e89de0714b6cf319448bfee1cf83d33c9f7bd6d439627f5/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/973fa4078033110d1cf17d12d7c74370d627c50584c08597c1dd0b031aeac820/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/973fa4078033110d1cf17d12d7c74370d627c50584c08597c1dd0b031aeac820/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98def0669601be1afa30b68afafc9f9bb6e173de5259bb5fd882e3667b492f09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98def0669601be1afa30b68afafc9f9bb6e173de5259bb5fd882e3667b492f09/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/acc06e41bfb0dde88ef8a71defe049f7f5046423248f71f9feded43539f333b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/acc06e41bfb0dde88ef8a71defe049f7f5046423248f71f9feded43539f333b1/userdata/shm major:0 minor:380 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5231174733af6dfccda009ede094b16d8605f1d544c8650e25602e453f823ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5231174733af6dfccda009ede094b16d8605f1d544c8650e25602e453f823ed/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8e96aa62ac1a645d730795eaee1426f93bf779657e28dfae2de6584d4698f47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8e96aa62ac1a645d730795eaee1426f93bf779657e28dfae2de6584d4698f47/userdata/shm major:0 minor:301 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca49fcd26f642fae0ad4b0e79ce89067828213cedc2ad2266d2e09ac67c2765b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca49fcd26f642fae0ad4b0e79ce89067828213cedc2ad2266d2e09ac67c2765b/userdata/shm major:0 minor:376 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbe23aee528bb0703b9f56eef4e9af48a51c00e2cae7c30ebf7b9a769eaa3ca5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbe23aee528bb0703b9f56eef4e9af48a51c00e2cae7c30ebf7b9a769eaa3ca5/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3fb8defa21a2e126f88bfe892784f4fe6d9ab5399e650c548c56991a01c8746/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3fb8defa21a2e126f88bfe892784f4fe6d9ab5399e650c548c56991a01c8746/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e435908e8c3b11d7a0a396cdbcc0b995c24c9229caf0774ce203615e976a7209/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e435908e8c3b11d7a0a396cdbcc0b995c24c9229caf0774ce203615e976a7209/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee2095a2270c3e367d66fc11cc6adb6430f788db587db7a903662543a24f764b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee2095a2270c3e367d66fc11cc6adb6430f788db587db7a903662543a24f764b/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5fa9216f177e87c477e1263a558f14e20de1af6f27bba84a2fe048320d3ff15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5fa9216f177e87c477e1263a558f14e20de1af6f27bba84a2fe048320d3ff15/userdata/shm major:0 minor:347 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd54fd7fa5dda74c458b8e324cb76b12e1d9da0704ee2114408db5a55439c272/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd54fd7fa5dda74c458b8e324cb76b12e1d9da0704ee2114408db5a55439c272/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d1b5f20-523b-4c79-a942-b0dfce8f64f9/volumes/kubernetes.io~projected/kube-api-access-4l295:{mountpoint:/var/lib/kubelet/pods/0d1b5f20-523b-4c79-a942-b0dfce8f64f9/volumes/kubernetes.io~projected/kube-api-access-4l295 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38dfb740-4469-4101-a309-4070aeb2dfe3/volumes/kubernetes.io~projected/kube-api-access-thwgf:{mountpoint:/var/lib/kubelet/pods/38dfb740-4469-4101-a309-4070aeb2dfe3/volumes/kubernetes.io~projected/kube-api-access-thwgf major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38dfb740-4469-4101-a309-4070aeb2dfe3/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/38dfb740-4469-4101-a309-4070aeb2dfe3/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38dfb740-4469-4101-a309-4070aeb2dfe3/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/38dfb740-4469-4101-a309-4070aeb2dfe3/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volume-subpaths/web-config/config-reloader/2 major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volume-subpaths/web-config/prometheus/15 major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~empty-dir/config-out major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~projected/kube-api-access-mm689:{mountpoint:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~projected/kube-api-access-mm689 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~projected/tls-assets major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/config major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/web-config major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b/volumes/kubernetes.io~projected/kube-api-access-c68dp:{mountpoint:/var/lib/kubelet/pods/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b/volumes/kubernetes.io~projected/kube-api-access-c68dp major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a381148-a1d6-404f-94ca-eed5b9481b7a/volumes/kubernetes.io~projected/kube-api-access-gtrgr:{mountpoint:/var/lib/kubelet/pods/5a381148-a1d6-404f-94ca-eed5b9481b7a/volumes/kubernetes.io~projected/kube-api-access-gtrgr major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a381148-a1d6-404f-94ca-eed5b9481b7a/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/5a381148-a1d6-404f-94ca-eed5b9481b7a/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a381148-a1d6-404f-94ca-eed5b9481b7a/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/5a381148-a1d6-404f-94ca-eed5b9481b7a/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6edb8e5f-e185-48be-8fed-5ac6899134fa/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6edb8e5f-e185-48be-8fed-5ac6899134fa/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/752e574a-9cd1-40a1-a65e-7b2aabd4434e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/752e574a-9cd1-40a1-a65e-7b2aabd4434e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/752e574a-9cd1-40a1-a65e-7b2aabd4434e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/752e574a-9cd1-40a1-a65e-7b2aabd4434e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/752e574a-9cd1-40a1-a65e-7b2aabd4434e/volumes/kubernetes.io~projected/kube-api-access-qckd7:{mountpoint:/var/lib/kubelet/pods/752e574a-9cd1-40a1-a65e-7b2aabd4434e/volumes/kubernetes.io~projected/kube-api-access-qckd7 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75ac2c49-e052-4613-a573-0a8cf1475ad3/volumes/kubernetes.io~projected/kube-api-access-q96hr:{mountpoint:/var/lib/kubelet/pods/75ac2c49-e052-4613-a573-0a8cf1475ad3/volumes/kubernetes.io~projected/kube-api-access-q96hr major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75ac2c49-e052-4613-a573-0a8cf1475ad3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/75ac2c49-e052-4613-a573-0a8cf1475ad3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/765947ed-ccf0-4296-aac8-eace5ce6de20/volumes/kubernetes.io~projected/kube-api-access-p6q45:{mountpoint:/var/lib/kubelet/pods/765947ed-ccf0-4296-aac8-eace5ce6de20/volumes/kubernetes.io~projected/kube-api-access-p6q45 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d6fdbfe-0357-4139-a034-28faec681123/volumes/kubernetes.io~projected/kube-api-access-srpcs:{mountpoint:/var/lib/kubelet/pods/7d6fdbfe-0357-4139-a034-28faec681123/volumes/kubernetes.io~projected/kube-api-access-srpcs major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86e95d56-5a28-4cdd-b23d-e56a55de786f/volumes/kubernetes.io~projected/kube-api-access-qbs5z:{mountpoint:/var/lib/kubelet/pods/86e95d56-5a28-4cdd-b23d-e56a55de786f/volumes/kubernetes.io~projected/kube-api-access-qbs5z major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86e95d56-5a28-4cdd-b23d-e56a55de786f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/86e95d56-5a28-4cdd-b23d-e56a55de786f/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ba702dc-08af-46e3-8235-61da0c303efd/volumes/kubernetes.io~projected/kube-api-access-5ndcg:{mountpoint:/var/lib/kubelet/pods/8ba702dc-08af-46e3-8235-61da0c303efd/volumes/kubernetes.io~projected/kube-api-access-5ndcg major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98c4af2a-2166-4fa0-99d0-662be28b6899/volumes/kubernetes.io~projected/kube-api-access-gb7k9:{mountpoint:/var/lib/kubelet/pods/98c4af2a-2166-4fa0-99d0-662be28b6899/volumes/kubernetes.io~projected/kube-api-access-gb7k9 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e8347f7-98bc-4826-a26d-2ad74ec8b27e/volumes/kubernetes.io~projected/kube-api-access-wl249:{mountpoint:/var/lib/kubelet/pods/9e8347f7-98bc-4826-a26d-2ad74ec8b27e/volumes/kubernetes.io~projected/kube-api-access-wl249 major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e8347f7-98bc-4826-a26d-2ad74ec8b27e/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/9e8347f7-98bc-4826-a26d-2ad74ec8b27e/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e8347f7-98bc-4826-a26d-2ad74ec8b27e/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/9e8347f7-98bc-4826-a26d-2ad74ec8b27e/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6/volumes/kubernetes.io~projected/kube-api-access-959fd:{mountpoint:/var/lib/kubelet/pods/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6/volumes/kubernetes.io~projected/kube-api-access-959fd major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca99b14b-c183-4966-85c2-b685be64e4cb/volumes/kubernetes.io~projected/kube-api-access-g26rs:{mountpoint:/var/lib/kubelet/pods/ca99b14b-c183-4966-85c2-b685be64e4cb/volumes/kubernetes.io~projected/kube-api-access-g26rs major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca99b14b-c183-4966-85c2-b685be64e4cb/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/ca99b14b-c183-4966-85c2-b685be64e4cb/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d31c1857-282f-4a04-89a9-883b1c2a7c19/volumes/kubernetes.io~projected/kube-api-access-grckl:{mountpoint:/var/lib/kubelet/pods/d31c1857-282f-4a04-89a9-883b1c2a7c19/volumes/kubernetes.io~projected/kube-api-access-grckl major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d31c1857-282f-4a04-89a9-883b1c2a7c19/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/d31c1857-282f-4a04-89a9-883b1c2a7c19/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d31c1857-282f-4a04-89a9-883b1c2a7c19/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/d31c1857-282f-4a04-89a9-883b1c2a7c19/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7f074e8-1737-4f5c-99cd-928d9c565c17/volumes/kubernetes.io~projected/kube-api-access-6hmwh:{mountpoint:/var/lib/kubelet/pods/d7f074e8-1737-4f5c-99cd-928d9c565c17/volumes/kubernetes.io~projected/kube-api-access-6hmwh major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7f074e8-1737-4f5c-99cd-928d9c565c17/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/d7f074e8-1737-4f5c-99cd-928d9c565c17/volumes/kubernetes.io~secret/metrics-certs major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da8029d1-50de-4612-adce-3a3f582c74d8/volumes/kubernetes.io~projected/kube-api-access-9f8gp:{mountpoint:/var/lib/kubelet/pods/da8029d1-50de-4612-adce-3a3f582c74d8/volumes/kubernetes.io~projected/kube-api-access-9f8gp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e50192cd-cb7b-4b3e-978f-661093207b81/volumes/kubernetes.io~projected/kube-api-access-l6ncm:{mountpoint:/var/lib/kubelet/pods/e50192cd-cb7b-4b3e-978f-661093207b81/volumes/kubernetes.io~projected/kube-api-access-l6ncm major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fea00aae-5dd9-48ef-b7d4-af08eed1481d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/fea00aae-5dd9-48ef-b7d4-af08eed1481d/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a4806507c3a7a626bb7cb944a8ed4ef2cb60e6df071f44720facfb416eb960bc/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/74701e5f9d29fb1ec1a7a805690d8a26840931cb61264166f085f3667a0775e5/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1a2077b8609d4d0e7b691666480cdb64e4c41cb6cbd0b1b2f4fdf42cf0ac430c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/b386f3a8e8a5160471ebe952087fd49db2febbfc9cce0a5c17ae566a26e1204c/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/fa9537e44d2509606ddc43c96e93306a9a5843150985e8d0bda62e4337d923fa/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/71047df82629cfb0c6c0afb6b42475db91f53244741c0daf414f6cd7e1d22629/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6c15adc008d0cae45a39ad2503991d96c16b8107918bd96bb6af185dec6b171e/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/5421f7d43e847e948cf6d9db8d3e22ba6ade97971998958c205ace43b9b7abf0/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ab51e8ad4268bae71fcdb3cea1a8f8b2250f2ff0995a042e7f9f176814e41c9e/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/8d03ffa5c7fc870dc6f3f63934255b133305903cf043063bedce7283ce70f08e/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/b520ad1565ed2ed9b61c0264d116036913b0c043a87bcdd5cf0769048fa582e8/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/280c18855819aba7a7a036f5eaca6bf6613d152e45403d11cc947acf78b0ff18/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/95c4373f7247d33298ea0b41418b4dcbdf32ff0ade1a34230f508cc9b336c8f8/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1740943cc2876e98935e73e86d7b5e904e3304cdd547086956bbb29c8cc72c0b/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/985ab946efe0e430e7aadc268df8cb96664eb5ddaaba9a6cf32921a0324f79db/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/30c5066210b4a0aacd0022f7572c39a9864a256fdbde14c798ff17e944d45ea5/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/1d3a555165481a71be8fe3c5d3374b97fc6383592af69e671865df95a4e23736/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/ada48db2b5e7a4339aa938d138575c45165b11ed783ddd68863805eaeb069dcb/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/41a342b083e33749af0862bbbb8a0345290c744a9b9abf8e66154a93ad46d68c/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/3a8e8a3f201fc84d192b6d2a46e43a93e96a1ff25736508ab28623d0788514fc/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/0773ad47a359efda8a0cd0c1b0a276b729424d97024c17eb10b9f78749525099/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/5d376273d7a821b2751d04d3165ac29e007f74686676f382b2b65d392b85fd58/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/5945393a5c70f722b642e03e4dc01c6552f6f4e8967cc7c77d636419cdc51b40/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/65f84b69da11aa6ec6596c4f6ca03332b02bac028c0397f027b50d88d35ce278/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/f9329c159b9e7d47ff4c6d70493961ea9b467d157ff62c5eb5619dff0cc0b808/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/5ae4b775a227253e2c8e954c6384f8be1b164de327d987fad70eedf27b8eef0e/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/d78fd2558670f187d719c574fbd00d2c2b8f97c787aaa39c7e5fa9a11228ffba/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/5cf81030393b28eba03110ea9cb3816906e5126a1b59c0031dde3ae4d5d58a60/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/19641334f29c0553e1c2506c7bf548185964ce2aca6bb055126d244599b029e6/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/7041df3b247819fb7cef6d35b9008e3ba49e122452b8a84303f0bd140b3c269e/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/c2dc24c7b98647c68b822602c7fda2fd57096fbc157e2851412d4f865863f836/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/a7fcd866479929bd418268865d2e4a590cc85af82f58f87b9dc75e929029badb/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/b7ae9e6fbcbcb63ff388180778c3d9858786f2a4d07ebe90faef1ec7bc679571/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/e1d96c004178db9b855caa21287dd6dd2ace8eab57b47a474c9d095f484eed9c/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/398994a600637037fd8edcf2d48671a3842334caca68e074d9435344725102c2/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/dbc4f73dd7684544c7ecf815e56b95f0f355787c18b621ba1a69b7d3ebc06f5c/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/5e031cbb22824a9bfbcb3503b5a29ee09c2d644cc89a7afe10251abad77ecd4e/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/273f2b692a0832cdb5326c333f9c9d2e19435d60e070580b285f1be2b56c73b1/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/a1bfeefdc31fc89ae0e77bde5d9bfc84e98fd8afc9cbfffdaf9007f8a2bbba9a/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/d1e7ee9350e2d30fa6aea30c71397548fff379c79e5ae7e014ce37a9cfb9f019/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/07b2ebe3b4ceda02d7ee712d3000d0f4731039ce594d37648e7c5d02b566e68c/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/0bdb9e5142df398c49011dd5ddab1324b5628036de38a855cf70273c70b2b4b5/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/e77f88d7ca0aa683223e9bf0e308d955bfb473530e5da8f4d95061158c076e68/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/502043b041f950e750f7ddc3c22e44e9dde900ba454b4a940907b2ecf0fa480e/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/da3b60877d3e300098c46468032e5d90f1dab655f3be897e16c3fab1580134d1/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/2f36486774038517c3ed192618911a6ac9a9bdc83c77b4bba6aa081c454763f2/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/a8c330f2984e04847e7d97e6e5bf32261c7029871e0811685ffca9eafd201479/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/ffdc467c9fdb866b4e7ea8058af6580c44ed362de8af483a80862e216a6d14e3/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/77a8c2cefae6a24bb84a85881b1ca1c26a3e241fc18e84c01b18d96e81574674/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/b2e63e4bdec59db6b0913b9ed34e44ae8bb21f9a023255c818b7c863ffcaaacd/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/611447f140a6b86d84390fb40e86a441ed4e262f9c28ad5de507b422fe0d3e1a/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/d49ec0d31fd0fb24248a727c9ba55066301f67a0f8b4e57ea061ad8c9923570e/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/48ac4ddaf1a4e50d1ebfc67dbfb0361c9ea66c377608a856c563aeba209ade55/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/cbb06b0d013c0e4d9d876dea878ac049f3f3c3da01f5f02353da916f2cfca429/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/c1570acf004908b31a31a188a9a3ab76f08dd73f7fd6a174dc4fed88b0ce0902/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/77eef3dd20ddbe7d53c86af5d8b3fb44963be0aea5365ff0f055119d86c4c593/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/eddd8c23ce1f90e8ffe937d76545e473dc93a69d16da2f6e65b70b680827b45b/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/a6c7fa1c3b5ab300eeaf079217ffadfafab5f19e53a22874ff817ba31a8dcd80/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/c7cdb0d6ecc4428ece30f89d2211c778feab6be9fc4b3df8774a0566646dedae/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/f034a990e01dd6aa0208b9e9c92ad1344416f79783d81086e1ac02ce6ab9787a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/78eb8a52529ee1e29368a19aa52b055067a5e1159f28ca9b803bde8f7d6a2d0b/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/9923e75b8dfba67924c088d283372f5ad0b172b74a124ea921adb301b669ad2f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/0e030b3bab34f0d34ad9f635765a226e04492febebacdfa3b89e3e65c31d1205/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/441096461e74800128fa9c7273c58d0ba59ec6f8a1508a2777db7ce05d5a0373/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/4fafb0491a9fcc6c1b75e8b83986e0a1b12d1dde21e4e1ef94ac06341294ee3f/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/309f55c43db8fac9334b4ee322c0d58f77ae1c74484c4bd52117267a305f04f4/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.719855    5428 manager.go:217] Machine: {Timestamp:2026-10-29 22:43:16.719057812 +0000 UTC m=+0.062006554 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093063 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26c30ad3e6218ba2bd0075085c8ebc SystemUUID:ec26c30a-d3e6-218b-a2bd-0075085c8ebc BootID:b936c840-e43d-407b-a4d9-55877095a3fd Filesystems:[{Device:/run/containers/storage/overlay-containers/fd54fd7fa5dda74c458b8e324cb76b12e1d9da0704ee2114408db5a55439c272/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3fb8defa21a2e126f88bfe892784f4fe6d9ab5399e650c548c56991a01c8746/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e435908e8c3b11d7a0a396cdbcc0b995c24c9229caf0774ce203615e976a7209/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca49fcd26f642fae0ad4b0e79ce89067828213cedc2ad2266d2e09ac67c2765b/userdata/shm DeviceMajor:0 DeviceMinor:376 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/752e574a-9cd1-40a1-a65e-7b2aabd4434e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee2095a2270c3e367d66fc11cc6adb6430f788db587db7a903662543a24f764b/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5231174733af6dfccda009ede094b16d8605f1d544c8650e25602e453f823ed/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:304 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58676504ac4c1ffb5167915e7395604a0c201844b2e26d83f32c3a64cc07a6f2/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e50192cd-cb7b-4b3e-978f-661093207b81/volumes/kubernetes.io~projected/kube-api-access-l6ncm DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/752e574a-9cd1-40a1-a65e-7b2aabd4434e/volumes/kubernetes.io~projected/kube-api-access-qckd7 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d7f074e8-1737-4f5c-99cd-928d9c565c17/volumes/kubernetes.io~projected/kube-api-access-6hmwh DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/75ac2c49-e052-4613-a573-0a8cf1475ad3/volumes/kubernetes.io~projected/kube-api-access-q96hr DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca99b14b-c183-4966-85c2-b685be64e4cb/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a381148-a1d6-404f-94ca-eed5b9481b7a/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:308 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ca99b14b-c183-4966-85c2-b685be64e4cb/volumes/kubernetes.io~projected/kube-api-access-g26rs DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~projected/kube-api-access-mm689 DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5fa9216f177e87c477e1263a558f14e20de1af6f27bba84a2fe048320d3ff15/userdata/shm DeviceMajor:0 DeviceMinor:347 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a381148-a1d6-404f-94ca-eed5b9481b7a/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:313 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/98c4af2a-2166-4fa0-99d0-662be28b6899/volumes/kubernetes.io~projected/kube-api-access-gb7k9 DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d54764a466fb9eb6014266167e8c7115df9fedebe5929c729e64a5fa96dbae3/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01e301c1d0db1e275053aa92117803e0d542ae1aba7e16cce13e6e629981bb46/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6edb8e5f-e185-48be-8fed-5ac6899134fa/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:367 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e8347f7-98bc-4826-a26d-2ad74ec8b27e/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:269 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8e96aa62ac1a645d730795eaee1426f93bf779657e28dfae2de6584d4698f47/userdata/shm DeviceMajor:0 DeviceMinor:301 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:303 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b/volumes/kubernetes.io~projected/kube-api-access-c68dp DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98def0669601be1afa30b68afafc9f9bb6e173de5259bb5fd882e3667b492f09/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01d7dbcf2939dfd219ce95104599895c03376407a6809ca7484bad8883735898/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/752e574a-9cd1-40a1-a65e-7b2aabd4434e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:306 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38dfb740-4469-4101-a309-4070aeb2dfe3/volumes/kubernetes.io~projected/kube-api-access-thwgf DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:310 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:311 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4041352b1166bf40c3c43c4d03741fc69b57864ae22033ff0386eca2c885d552/userdata/shm DeviceMajor:0 DeviceMinor:315 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/765947ed-ccf0-4296-aac8-eace5ce6de20/volumes/kubernetes.io~projected/kube-api-access-p6q45 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78f01703269066ee6b5be698312e3856eee8652f2fbbf9dcf4f145eb25743e8a/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e8347f7-98bc-4826-a26d-2ad74ec8b27e/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:265 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a381148-a1d6-404f-94ca-eed5b9481b7a/volumes/kubernetes.io~projected/kube-api-access-gtrgr DeviceMajor:0 DeviceMinor:300 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32ad4b291e006e99e6a5924e04a9322cd4d78b5f2c43c9fd409adbf6a52e0a3d/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/acc06e41bfb0dde88ef8a71defe049f7f5046423248f71f9feded43539f333b1/userdata/shm DeviceMajor:0 DeviceMinor:380 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/75ac2c49-e052-4613-a573-0a8cf1475ad3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/38dfb740-4469-4101-a309-4070aeb2dfe3/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:309 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d1b5f20-523b-4c79-a942-b0dfce8f64f9/volumes/kubernetes.io~projected/kube-api-access-4l295 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38dfb740-4469-4101-a309-4070aeb2dfe3/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbe23aee528bb0703b9f56eef4e9af48a51c00e2cae7c30ebf7b9a769eaa3ca5/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ba702dc-08af-46e3-8235-61da0c303efd/volumes/kubernetes.io~projected/kube-api-access-5ndcg DeviceMajor:0 DeviceMinor:375 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/da8029d1-50de-4612-adce-3a3f582c74d8/volumes/kubernetes.io~projected/kube-api-access-9f8gp DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fea00aae-5dd9-48ef-b7d4-af08eed1481d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e5cbc8c2f59c08d0e89de0714b6cf319448bfee1cf83d33c9f7bd6d439627f5/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d7f074e8-1737-4f5c-99cd-928d9c565c17/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:372 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6/volumes/kubernetes.io~projected/kube-api-access-959fd DeviceMajor:0 DeviceMinor:277 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:307 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/644826cb83d272dc432199658b25ed68b6ac98d886cf927f903de2a1447fcb93/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86e95d56-5a28-4cdd-b23d-e56a55de786f/volumes/kubernetes.io~projected/kube-api-access-qbs5z DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11dab8a95c8e0b059d8aa2c664a8555df7364d89df659eeade4c6d8a0c6e7a7e/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/973fa4078033110d1cf17d12d7c74370d627c50584c08597c1dd0b031aeac820/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d31c1857-282f-4a04-89a9-883b1c2a7c19/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e8347f7-98bc-4826-a26d-2ad74ec8b27e/volumes/kubernetes.io~projected/kube-api-access-wl249 DeviceMajor:0 DeviceMinor:270 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:305 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7d6fdbfe-0357-4139-a034-28faec681123/volumes/kubernetes.io~projected/kube-api-access-srpcs DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86e95d56-5a28-4cdd-b23d-e56a55de786f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40fb2dc7-336c-4483-93e7-68309e345b22/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d31c1857-282f-4a04-89a9-883b1c2a7c19/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d31c1857-282f-4a04-89a9-883b1c2a7c19/volumes/kubernetes.io~projected/kube-api-access-grckl DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:01e301c1d0db1e2 MacAddress:96:64:0d:87:83:8b Speed:10000 Mtu:8901} {Name:11dab8a95c8e0b0 MacAddress:ee:7f:95:a5:d9:8a Speed:10000 Mtu:8901} {Name:4041352b1166bf4 MacAddress:ce:63:f8:c2:a2:c9 Speed:10000 Mtu:8901} {Name:8e5cbc8c2f59c08 MacAddress:b2:8b:09:c6:20:ed Speed:10000 Mtu:8901} {Name:973fa4078033110 MacAddress:36:88:1d:d5:77:70 Speed:10000 Mtu:8901} {Name:acc06e41bfb0dde MacAddress:2e:f6:31:a9:b5:a6 Speed:10000 Mtu:8901} {Name:b5231174733af6d MacAddress:ee:25:1e:66:84:05 Speed:10000 Mtu:8901} {Name:b8e96aa62ac1a64 MacAddress:be:af:15:76:87:b9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:da:3c:e2:17 Speed:0 Mtu:9001} {Name:br-int MacAddress:1e:b1:fc:c2:51:35 Speed:0 Mtu:8901} {Name:ca49fcd26f642fa MacAddress:8e:8c:03:6f:81:e1 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:da:3c:e2:17 Speed:0 Mtu:9001} {Name:f5fa9216f177e87 MacAddress:1a:04:36:12:b7:64 Speed:10000 Mtu:8901} {Name:fd54fd7fa5dda74 MacAddress:de:f3:26:28:eb:04 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:56:24:18:82:fd:d8 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1a:1e:8b:0e:51:18 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.720503    5428 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.720562    5428 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.720840    5428 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.720860    5428 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-127.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.720975    5428 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.720983    5428 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.720993    5428 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.721012    5428 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.721079    5428 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.721225    5428 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.721322    5428 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.721353    5428 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.721366    5428 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.721375    5428 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.721384    5428 kubelet.go:405] "Adding apiserver pod source"
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.721392    5428 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.722023    5428 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.722033    5428 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:43:16.723569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.722463    5428 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 22:43:16.724730 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.724716    5428 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 22:43:16.724761 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.724731    5428 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:43:16.724854 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.724843    5428 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 29 22:43:16.725025 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.725015    5428 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 22:43:16.725054 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.725044    5428 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 22:43:16.725154 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.725145    5428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 22:43:16.725190 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.725160    5428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 22:43:16.725190 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.725166    5428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 22:43:16.725190 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.725172    5428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 22:43:16.725190 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.725181    5428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 22:43:16.725190 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.725190    5428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 22:43:16.725323 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.725205    5428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 22:43:16.725323 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.725211    5428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 22:43:16.725323 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.725218    5428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 22:43:16.725323 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.725224    5428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 22:43:16.725323 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.725237    5428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 22:43:16.725323 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.725246    5428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 22:43:16.725323 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.725270    5428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 22:43:16.725323 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.725279    5428 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 22:43:16.728507 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.728493    5428 server.go:1263] "Started kubelet"
Oct 29 22:43:16.728585 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.728551    5428 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 22:43:16.728636 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.728573    5428 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 22:43:16.729000 ip-10-0-134-127 systemd[1]: Started Kubernetes Kubelet.
Oct 29 22:43:16.729224 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.729205    5428 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 22:43:16.729523 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.729499    5428 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 22:43:16.733432 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.733407    5428 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 22:43:16.735148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.733525    5428 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 22:43:16.735148 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:43:16.733636    5428 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-127.ec2.internal\" not found"
Oct 29 22:43:16.735148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.733717    5428 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 22:43:16.735148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.733731    5428 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 22:43:16.735148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.733853    5428 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 22:43:16.735148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.733871    5428 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 22:43:16.735148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.734649    5428 factory.go:55] Registering systemd factory
Oct 29 22:43:16.735148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.734668    5428 factory.go:223] Registration of the systemd container factory successfully
Oct 29 22:43:16.735148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.734747    5428 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:37:07 +0000 UTC" deadline="2028-06-27 06:22:38.679893273 +0000 UTC"
Oct 29 22:43:16.735148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.734787    5428 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14551h39m21.945110581s"
Oct 29 22:43:16.735148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.734948    5428 factory.go:153] Registering CRI-O factory
Oct 29 22:43:16.735148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.734965    5428 factory.go:223] Registration of the crio container factory successfully
Oct 29 22:43:16.735148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.735049    5428 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 22:43:16.735148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.735069    5428 factory.go:103] Registering Raw factory
Oct 29 22:43:16.735148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.735087    5428 manager.go:1199] Started watching for new ooms in manager
Oct 29 22:43:16.736083 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.736067    5428 manager.go:319] Starting recovery of all containers
Oct 29 22:43:16.737132 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737098    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d31c1857-282f-4a04-89a9-883b1c2a7c19" volumeName="kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-oauth-config" seLinuxMountContext=""
Oct 29 22:43:16.737201 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737138    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 29 22:43:16.737201 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737156    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 29 22:43:16.737201 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737172    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d31c1857-282f-4a04-89a9-883b1c2a7c19" volumeName="kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-serving-cert" seLinuxMountContext=""
Oct 29 22:43:16.737201 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737187    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98c4af2a-2166-4fa0-99d0-662be28b6899" volumeName="kubernetes.io/configmap/98c4af2a-2166-4fa0-99d0-662be28b6899-cni-binary-copy" seLinuxMountContext=""
Oct 29 22:43:16.737465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737201    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d31c1857-282f-4a04-89a9-883b1c2a7c19" volumeName="kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-config" seLinuxMountContext=""
Oct 29 22:43:16.737465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737213    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="752e574a-9cd1-40a1-a65e-7b2aabd4434e" volumeName="kubernetes.io/empty-dir/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-tuned" seLinuxMountContext=""
Oct 29 22:43:16.737465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737226    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38dfb740-4469-4101-a309-4070aeb2dfe3" volumeName="kubernetes.io/configmap/38dfb740-4469-4101-a309-4070aeb2dfe3-metrics-client-ca" seLinuxMountContext=""
Oct 29 22:43:16.737465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737241    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38dfb740-4469-4101-a309-4070aeb2dfe3" volumeName="kubernetes.io/secret/38dfb740-4469-4101-a309-4070aeb2dfe3-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 29 22:43:16.737465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737253    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 29 22:43:16.737465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737267    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 29 22:43:16.737465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737281    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-web-config" seLinuxMountContext=""
Oct 29 22:43:16.737465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737293    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a381148-a1d6-404f-94ca-eed5b9481b7a" volumeName="kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-oauth-config" seLinuxMountContext=""
Oct 29 22:43:16.737465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737306    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e8347f7-98bc-4826-a26d-2ad74ec8b27e" volumeName="kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-trusted-ca-bundle" seLinuxMountContext=""
Oct 29 22:43:16.737465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737323    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d1b5f20-523b-4c79-a942-b0dfce8f64f9" volumeName="kubernetes.io/configmap/0d1b5f20-523b-4c79-a942-b0dfce8f64f9-iptables-alerter-script" seLinuxMountContext=""
Oct 29 22:43:16.737465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737335    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38dfb740-4469-4101-a309-4070aeb2dfe3" volumeName="kubernetes.io/secret/38dfb740-4469-4101-a309-4070aeb2dfe3-prometheus-operator-tls" seLinuxMountContext=""
Oct 29 22:43:16.737465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737349    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 29 22:43:16.737465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737362    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a381148-a1d6-404f-94ca-eed5b9481b7a" volumeName="kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-trusted-ca-bundle" seLinuxMountContext=""
Oct 29 22:43:16.737465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737376    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e8347f7-98bc-4826-a26d-2ad74ec8b27e" volumeName="kubernetes.io/empty-dir/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-tmp" seLinuxMountContext=""
Oct 29 22:43:16.737465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737388    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d31c1857-282f-4a04-89a9-883b1c2a7c19" volumeName="kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-service-ca" seLinuxMountContext=""
Oct 29 22:43:16.737465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737400    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d31c1857-282f-4a04-89a9-883b1c2a7c19" volumeName="kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-trusted-ca-bundle" seLinuxMountContext=""
Oct 29 22:43:16.737465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737412    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d31c1857-282f-4a04-89a9-883b1c2a7c19" volumeName="kubernetes.io/projected/d31c1857-282f-4a04-89a9-883b1c2a7c19-kube-api-access-grckl" seLinuxMountContext=""
Oct 29 22:43:16.737465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737431    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/projected/40fb2dc7-336c-4483-93e7-68309e345b22-tls-assets" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737444    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a381148-a1d6-404f-94ca-eed5b9481b7a" volumeName="kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-service-ca" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737539    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d31c1857-282f-4a04-89a9-883b1c2a7c19" volumeName="kubernetes.io/empty-dir/d31c1857-282f-4a04-89a9-883b1c2a7c19-tmp" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737552    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7f074e8-1737-4f5c-99cd-928d9c565c17" volumeName="kubernetes.io/projected/d7f074e8-1737-4f5c-99cd-928d9c565c17-kube-api-access-6hmwh" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737565    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e50192cd-cb7b-4b3e-978f-661093207b81" volumeName="kubernetes.io/projected/e50192cd-cb7b-4b3e-978f-661093207b81-kube-api-access-l6ncm" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737578    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a381148-a1d6-404f-94ca-eed5b9481b7a" volumeName="kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-serving-cert" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737597    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="752e574a-9cd1-40a1-a65e-7b2aabd4434e" volumeName="kubernetes.io/projected/752e574a-9cd1-40a1-a65e-7b2aabd4434e-kube-api-access-qckd7" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737610    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d6fdbfe-0357-4139-a034-28faec681123" volumeName="kubernetes.io/configmap/7d6fdbfe-0357-4139-a034-28faec681123-multus-daemon-config" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737634    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75ac2c49-e052-4613-a573-0a8cf1475ad3" volumeName="kubernetes.io/configmap/75ac2c49-e052-4613-a573-0a8cf1475ad3-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737648    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75ac2c49-e052-4613-a573-0a8cf1475ad3" volumeName="kubernetes.io/secret/75ac2c49-e052-4613-a573-0a8cf1475ad3-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737661    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737674    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-grpc-tls" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737688    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-metrics-client-certs" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737700    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737712    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="501b61a4-5a18-4c9f-8b37-0b162c9c2c2b" volumeName="kubernetes.io/configmap/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-ovnkube-config" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737725    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="501b61a4-5a18-4c9f-8b37-0b162c9c2c2b" volumeName="kubernetes.io/configmap/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-env-overrides" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737737    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6edb8e5f-e185-48be-8fed-5ac6899134fa" volumeName="kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737749    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75ac2c49-e052-4613-a573-0a8cf1475ad3" volumeName="kubernetes.io/empty-dir/75ac2c49-e052-4613-a573-0a8cf1475ad3-data-volume" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737762    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a381148-a1d6-404f-94ca-eed5b9481b7a" volumeName="kubernetes.io/empty-dir/5a381148-a1d6-404f-94ca-eed5b9481b7a-tmp" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737775    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e8347f7-98bc-4826-a26d-2ad74ec8b27e" volumeName="kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-oauth-serving-cert" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737788    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e8347f7-98bc-4826-a26d-2ad74ec8b27e" volumeName="kubernetes.io/projected/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-kube-api-access-wl249" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737816    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737830    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/empty-dir/40fb2dc7-336c-4483-93e7-68309e345b22-config-out" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737842    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-config" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737856    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75ac2c49-e052-4613-a573-0a8cf1475ad3" volumeName="kubernetes.io/projected/75ac2c49-e052-4613-a573-0a8cf1475ad3-kube-api-access-q96hr" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737868    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98c4af2a-2166-4fa0-99d0-662be28b6899" volumeName="kubernetes.io/projected/98c4af2a-2166-4fa0-99d0-662be28b6899-kube-api-access-gb7k9" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737880    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c44b4b5f-8aa2-4cf2-b72a-16b5002260b6" volumeName="kubernetes.io/projected/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-kube-api-access-959fd" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737892    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca99b14b-c183-4966-85c2-b685be64e4cb" volumeName="kubernetes.io/configmap/ca99b14b-c183-4966-85c2-b685be64e4cb-config-volume" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737904    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e50192cd-cb7b-4b3e-978f-661093207b81" volumeName="kubernetes.io/empty-dir/e50192cd-cb7b-4b3e-978f-661093207b81-tmp-dir" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737917    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="501b61a4-5a18-4c9f-8b37-0b162c9c2c2b" volumeName="kubernetes.io/secret/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737931    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca99b14b-c183-4966-85c2-b685be64e4cb" volumeName="kubernetes.io/projected/ca99b14b-c183-4966-85c2-b685be64e4cb-kube-api-access-g26rs" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737943    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7f074e8-1737-4f5c-99cd-928d9c565c17" volumeName="kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737955    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fea00aae-5dd9-48ef-b7d4-af08eed1481d" volumeName="kubernetes.io/configmap/fea00aae-5dd9-48ef-b7d4-af08eed1481d-konnectivity-ca" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737967    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fea00aae-5dd9-48ef-b7d4-af08eed1481d" volumeName="kubernetes.io/secret/fea00aae-5dd9-48ef-b7d4-af08eed1481d-agent-certs" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.737988    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738001    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 29 22:43:16.738007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738013    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738026    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="501b61a4-5a18-4c9f-8b37-0b162c9c2c2b" volumeName="kubernetes.io/configmap/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-ovnkube-script-lib" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738039    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a381148-a1d6-404f-94ca-eed5b9481b7a" volumeName="kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-config" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738051    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca99b14b-c183-4966-85c2-b685be64e4cb" volumeName="kubernetes.io/empty-dir/ca99b14b-c183-4966-85c2-b685be64e4cb-tmp-dir" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738062    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d31c1857-282f-4a04-89a9-883b1c2a7c19" volumeName="kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-oauth-serving-cert" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738074    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da8029d1-50de-4612-adce-3a3f582c74d8" volumeName="kubernetes.io/configmap/da8029d1-50de-4612-adce-3a3f582c74d8-serviceca" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738088    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/projected/40fb2dc7-336c-4483-93e7-68309e345b22-kube-api-access-mm689" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738099    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86e95d56-5a28-4cdd-b23d-e56a55de786f" volumeName="kubernetes.io/secret/86e95d56-5a28-4cdd-b23d-e56a55de786f-cert" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738111    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e8347f7-98bc-4826-a26d-2ad74ec8b27e" volumeName="kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-service-ca" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738123    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e8347f7-98bc-4826-a26d-2ad74ec8b27e" volumeName="kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-serving-cert" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738135    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca99b14b-c183-4966-85c2-b685be64e4cb" volumeName="kubernetes.io/secret/ca99b14b-c183-4966-85c2-b685be64e4cb-metrics-tls" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738149    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da8029d1-50de-4612-adce-3a3f582c74d8" volumeName="kubernetes.io/projected/da8029d1-50de-4612-adce-3a3f582c74d8-kube-api-access-9f8gp" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738162    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738173    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="765947ed-ccf0-4296-aac8-eace5ce6de20" volumeName="kubernetes.io/projected/765947ed-ccf0-4296-aac8-eace5ce6de20-kube-api-access-p6q45" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738185    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86e95d56-5a28-4cdd-b23d-e56a55de786f" volumeName="kubernetes.io/projected/86e95d56-5a28-4cdd-b23d-e56a55de786f-kube-api-access-qbs5z" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738198    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e8347f7-98bc-4826-a26d-2ad74ec8b27e" volumeName="kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-oauth-config" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738213    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="752e574a-9cd1-40a1-a65e-7b2aabd4434e" volumeName="kubernetes.io/empty-dir/752e574a-9cd1-40a1-a65e-7b2aabd4434e-tmp" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738224    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d6fdbfe-0357-4139-a034-28faec681123" volumeName="kubernetes.io/configmap/7d6fdbfe-0357-4139-a034-28faec681123-cni-binary-copy" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738244    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38dfb740-4469-4101-a309-4070aeb2dfe3" volumeName="kubernetes.io/projected/38dfb740-4469-4101-a309-4070aeb2dfe3-kube-api-access-thwgf" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738256    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="501b61a4-5a18-4c9f-8b37-0b162c9c2c2b" volumeName="kubernetes.io/projected/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-kube-api-access-c68dp" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738268    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a381148-a1d6-404f-94ca-eed5b9481b7a" volumeName="kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-oauth-serving-cert" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738280    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a381148-a1d6-404f-94ca-eed5b9481b7a" volumeName="kubernetes.io/projected/5a381148-a1d6-404f-94ca-eed5b9481b7a-kube-api-access-gtrgr" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738291    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e8347f7-98bc-4826-a26d-2ad74ec8b27e" volumeName="kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-config" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738303    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c44b4b5f-8aa2-4cf2-b72a-16b5002260b6" volumeName="kubernetes.io/configmap/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738315    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d1b5f20-523b-4c79-a942-b0dfce8f64f9" volumeName="kubernetes.io/projected/0d1b5f20-523b-4c79-a942-b0dfce8f64f9-kube-api-access-4l295" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738326    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98c4af2a-2166-4fa0-99d0-662be28b6899" volumeName="kubernetes.io/configmap/98c4af2a-2166-4fa0-99d0-662be28b6899-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738339    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c44b4b5f-8aa2-4cf2-b72a-16b5002260b6" volumeName="kubernetes.io/empty-dir/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-textfile" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738351    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c44b4b5f-8aa2-4cf2-b72a-16b5002260b6" volumeName="kubernetes.io/secret/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-tls" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738361    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c44b4b5f-8aa2-4cf2-b72a-16b5002260b6" volumeName="kubernetes.io/secret/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738372    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fb2dc7-336c-4483-93e7-68309e345b22" volumeName="kubernetes.io/empty-dir/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-db" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738385    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d6fdbfe-0357-4139-a034-28faec681123" volumeName="kubernetes.io/projected/7d6fdbfe-0357-4139-a034-28faec681123-kube-api-access-srpcs" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738397    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ba702dc-08af-46e3-8235-61da0c303efd" volumeName="kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738411    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98c4af2a-2166-4fa0-99d0-662be28b6899" volumeName="kubernetes.io/configmap/98c4af2a-2166-4fa0-99d0-662be28b6899-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738423    5428 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c44b4b5f-8aa2-4cf2-b72a-16b5002260b6" volumeName="kubernetes.io/configmap/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-metrics-client-ca" seLinuxMountContext=""
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738436    5428 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 22:43:16.739019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.738445    5428 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 22:43:16.767084 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.767049    5428 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 22:43:16.768445 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.768414    5428 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 22:43:16.768445 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.768440    5428 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 22:43:16.768645 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.768460    5428 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 22:43:16.768645 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:43:16.768497    5428 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 29 22:43:16.779567 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.779539    5428 generic.go:356] "Generic (PLEG): container finished" podID="404a57dfae016c255d65ef53a28af2de" containerID="15487a37e5318be4b7de51ac5bec7c3419861439da83418fe53d4158ec0490d3" exitCode=0
Oct 29 22:43:16.781728 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.781707    5428 generic.go:356] "Generic (PLEG): container finished" podID="c44b4b5f-8aa2-4cf2-b72a-16b5002260b6" containerID="116c137eac71033dda759df4da0b1efbd98f12bd958756467d13198fc82d481f" exitCode=0
Oct 29 22:43:16.782104 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.782084    5428 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 29 22:43:16.783398 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.783374    5428 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:43:16.783733 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.783711    5428 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:43:16.791658 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.791635    5428 generic.go:356] "Generic (PLEG): container finished" podID="40fb2dc7-336c-4483-93e7-68309e345b22" containerID="25def503b46771084aa879d81063461ce12224e245e01a5f16363e7163376790" exitCode=0
Oct 29 22:43:16.794484 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.794460    5428 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:43:16.802983 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.802961    5428 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:43:16.815924 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.815900    5428 generic.go:356] "Generic (PLEG): container finished" podID="98c4af2a-2166-4fa0-99d0-662be28b6899" containerID="85c841212b51d71aad4c4cabf9928c8517ed8f52337dfe0e38707bee0da36efe" exitCode=0
Oct 29 22:43:16.815924 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.815923    5428 generic.go:356] "Generic (PLEG): container finished" podID="98c4af2a-2166-4fa0-99d0-662be28b6899" containerID="75023404ed159ccdd0a7c0601a0c9deac7792df6af04c2752bc84281fa1b6aea" exitCode=0
Oct 29 22:43:16.816018 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.815934    5428 generic.go:356] "Generic (PLEG): container finished" podID="98c4af2a-2166-4fa0-99d0-662be28b6899" containerID="a31ab00e9e3e01baec8d093ecb70fcf6004686ea259d597013513f878b2aaa20" exitCode=0
Oct 29 22:43:16.816018 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.815941    5428 generic.go:356] "Generic (PLEG): container finished" podID="98c4af2a-2166-4fa0-99d0-662be28b6899" containerID="0ff4ee435fdde1c564f59980fc82bc74cfb10c7062ba8f85e349f80974e47a3d" exitCode=0
Oct 29 22:43:16.816018 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.815950    5428 generic.go:356] "Generic (PLEG): container finished" podID="98c4af2a-2166-4fa0-99d0-662be28b6899" containerID="f565f4b0e1303c8fb77d771b1be121537fc2daca4cd71d7457aef55a7cf642b9" exitCode=0
Oct 29 22:43:16.816018 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.815955    5428 generic.go:356] "Generic (PLEG): container finished" podID="98c4af2a-2166-4fa0-99d0-662be28b6899" containerID="51170e1aac8c0b00d08c7f80bbaa4cadc2bc1ea1b5e8bc8fe97604f55af5bfa7" exitCode=0
Oct 29 22:43:16.833241 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.833218    5428 manager.go:324] Recovery completed
Oct 29 22:43:16.855182 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.855119    5428 cpu_manager.go:226] "Starting" policy="none"
Oct 29 22:43:16.855182 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.855135    5428 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 22:43:16.855182 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.855152    5428 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 22:43:16.855380 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.855262    5428 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 29 22:43:16.855380 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.855273    5428 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 29 22:43:16.855380 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.855289    5428 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 29 22:43:16.855380 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.855294    5428 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 29 22:43:16.855380 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.855300    5428 policy_none.go:50] "Start"
Oct 29 22:43:16.855380 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.855308    5428 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 22:43:16.855380 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.855317    5428 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 22:43:16.855565 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.855434    5428 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 29 22:43:16.855565 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.855441    5428 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 29 22:43:16.855565 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.855446    5428 policy_none.go:44] "Start"
Oct 29 22:43:16.859691 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.859658    5428 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 22:43:16.863122 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.863105    5428 manager.go:341] "Starting Device Plugin manager"
Oct 29 22:43:16.863604 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:43:16.863131    5428 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 22:43:16.863604 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.863604    5428 server.go:87] "Starting device plugin registration server"
Oct 29 22:43:16.863920 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.863906    5428 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 22:43:16.863990 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.863917    5428 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 22:43:16.864042 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.863994    5428 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 22:43:16.864102 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.864082    5428 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 22:43:16.864151 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.864110    5428 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 22:43:16.864151 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.864116    5428 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 22:43:16.864249 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.864162    5428 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:43:16.864101007Z","UUID":"42b0df08-8c45-40d7-a2ac-d4bf1bd7c152","Handler":null,"Name":"","Endpoint":""}
Oct 29 22:43:16.865491 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.865473    5428 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 22:43:16.865593 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.865500    5428 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 22:43:16.865786 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:43:16.865767    5428 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 22:43:16.868626 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.868594    5428 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal"]
Oct 29 22:43:16.869252 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.869213    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal" event={"ID":"404a57dfae016c255d65ef53a28af2de","Type":"ContainerStarted","Data":"b776ecba16dec7e42d9d78a3c20459dcac252ca53033112d8df7914f960780cd"}
Oct 29 22:43:16.869252 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.869245    5428 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:43:16.869390 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.869257    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal" event={"ID":"404a57dfae016c255d65ef53a28af2de","Type":"ContainerDied","Data":"15487a37e5318be4b7de51ac5bec7c3419861439da83418fe53d4158ec0490d3"}
Oct 29 22:43:16.869390 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.869280    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal" event={"ID":"404a57dfae016c255d65ef53a28af2de","Type":"ContainerStarted","Data":"58676504ac4c1ffb5167915e7395604a0c201844b2e26d83f32c3a64cc07a6f2"}
Oct 29 22:43:16.869390 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.869372    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal" event={"ID":"a47eb3de0ec2692f9f0b4d5a8aa019c1","Type":"ContainerStarted","Data":"2b28a001751eccc381bca7e345083a9d4579d7a556e6d367ece8a55387f4a5c3"}
Oct 29 22:43:16.869514 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.869394    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal" event={"ID":"a47eb3de0ec2692f9f0b4d5a8aa019c1","Type":"ContainerStarted","Data":"78f01703269066ee6b5be698312e3856eee8652f2fbbf9dcf4f145eb25743e8a"}
Oct 29 22:43:16.869514 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.869434    5428 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal"
Oct 29 22:43:16.893323 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.893300    5428 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:43:16.893411 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:43:16.893344    5428 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-127.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal"
Oct 29 22:43:16.893950 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.893933    5428 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:43:16.894024 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:43:16.893976    5428 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:43:16.964354 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.964332    5428 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:43:16.966303 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.966286    5428 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:43:16.966375 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.966321    5428 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:43:16.966375 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.966331    5428 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-127.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:43:16.966375 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.966374    5428 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-127.ec2.internal"
Oct 29 22:43:16.983339 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.983314    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/404a57dfae016c255d65ef53a28af2de-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal\" (UID: \"404a57dfae016c255d65ef53a28af2de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:43:16.983412 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.983344    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a47eb3de0ec2692f9f0b4d5a8aa019c1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-127.ec2.internal\" (UID: \"a47eb3de0ec2692f9f0b4d5a8aa019c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal"
Oct 29 22:43:16.983412 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.983368    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/404a57dfae016c255d65ef53a28af2de-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal\" (UID: \"404a57dfae016c255d65ef53a28af2de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:43:16.997155 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.997138    5428 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-127.ec2.internal"
Oct 29 22:43:16.997194 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:16.997189    5428 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-127.ec2.internal"
Oct 29 22:43:17.084286 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.084256    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/404a57dfae016c255d65ef53a28af2de-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal\" (UID: \"404a57dfae016c255d65ef53a28af2de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:43:17.084440 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.084325    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/404a57dfae016c255d65ef53a28af2de-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal\" (UID: \"404a57dfae016c255d65ef53a28af2de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:43:17.084440 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.084361    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/404a57dfae016c255d65ef53a28af2de-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal\" (UID: \"404a57dfae016c255d65ef53a28af2de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:43:17.084440 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.084391    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a47eb3de0ec2692f9f0b4d5a8aa019c1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-127.ec2.internal\" (UID: \"a47eb3de0ec2692f9f0b4d5a8aa019c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal"
Oct 29 22:43:17.084440 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.084429    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a47eb3de0ec2692f9f0b4d5a8aa019c1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-127.ec2.internal\" (UID: \"a47eb3de0ec2692f9f0b4d5a8aa019c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal"
Oct 29 22:43:17.084568 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.084447    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/404a57dfae016c255d65ef53a28af2de-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal\" (UID: \"404a57dfae016c255d65ef53a28af2de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:43:17.722393 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.722362    5428 apiserver.go:52] "Watching apiserver"
Oct 29 22:43:17.767753 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.767717    5428 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 22:43:17.768321 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.768296    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal","openshift-console/console-6bff8847db-vg5bs","openshift-insights/insights-runtime-extractor-gpthh","openshift-console/console-774f56585c-2x9nc","openshift-multus/network-metrics-daemon-prbcb","openshift-monitoring/prometheus-k8s-0","openshift-dns/node-resolver-q8pg5","openshift-ingress-canary/ingress-canary-dgv4b","kube-system/global-pull-secret-syncer-qmfpg","openshift-cluster-node-tuning-operator/tuned-4bgs7","openshift-console/console-56797698b7-xlsbd","openshift-monitoring/prometheus-operator-9cd89b74c-87hkr","openshift-multus/multus-additional-cni-plugins-mlsbk","openshift-ovn-kubernetes/ovnkube-node-4xc2d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666","openshift-image-registry/node-ca-n27f7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal","openshift-monitoring/node-exporter-5pzlf","openshift-multus/multus-8dtlv","openshift-dns/dns-default-dcx8f","kube-system/konnectivity-agent-5jn2n","openshift-network-diagnostics/network-check-target-lqjl6","openshift-network-operator/iptables-alerter-2q4dp"]
Oct 29 22:43:17.783555 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.783522    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tj892\""
Oct 29 22:43:17.783722 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.783602    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 22:43:17.783722 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.783689    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 22:43:17.783722 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.783689    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 22:43:17.785906 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.785862    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mcv8v\""
Oct 29 22:43:17.786119 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786101    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 22:43:17.786215 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786193    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kfq4\""
Oct 29 22:43:17.786326 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786248    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-fchfg\""
Oct 29 22:43:17.786326 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786323    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 22:43:17.786437 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786336    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m52c5\""
Oct 29 22:43:17.786514 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786457    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 22:43:17.786574 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786519    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 22:43:17.786574 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786559    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rpfjl\""
Oct 29 22:43:17.786666 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786585    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 22:43:17.786666 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786649    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 22:43:17.786763 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786734    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 22:43:17.786910 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786770    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 22:43:17.786910 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786786    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 22:43:17.786910 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786795    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 22:43:17.786910 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786868    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h2plp\""
Oct 29 22:43:17.786910 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786902    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 22:43:17.787147 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786871    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 22:43:17.787147 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786918    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 22:43:17.787147 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786914    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 22:43:17.787147 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.786989    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 22:43:17.787147 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.787011    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 22:43:17.787147 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.787082    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 22:43:17.787323 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.787163    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 22:43:17.790414 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.790393    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 22:43:17.790525 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.790437    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 22:43:17.791851 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.791837    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 22:43:17.791943 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.791923    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 22:43:17.792061 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.792046    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 22:43:17.792107 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.792069    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 22:43:17.792707 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.792692    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 22:43:17.792844 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.792738    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:43:17.792844 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.792743    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 22:43:17.792844 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.792780    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:43:17.792987 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.792913    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ltz9s\""
Oct 29 22:43:17.792987 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.792958    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vllbq\""
Oct 29 22:43:17.792987 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.792980    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 22:43:17.793090 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.792990    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 22:43:17.795373 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.795349    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 22:43:17.795466 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.795391    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 22:43:17.795466 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.795423    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-28brx\""
Oct 29 22:43:17.795466 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.795424    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-phcjr\""
Oct 29 22:43:17.795606 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.795531    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 29 22:43:17.795667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.795613    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xjmd7\""
Oct 29 22:43:17.795667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.795645    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 22:43:17.795841 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.795722    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 22:43:17.795841 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.795725    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nxz4d\""
Oct 29 22:43:17.795841 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.795827    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 29 22:43:17.797869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.797850    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 22:43:17.823530 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.823513    5428 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal"
Oct 29 22:43:17.823653 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.823641    5428 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:43:17.840129 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.840104    5428 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 22:43:17.845766 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.845746    5428 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:43:17.845873 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:43:17.845794    5428 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-127.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-127.ec2.internal"
Oct 29 22:43:17.846393 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.846378    5428 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:43:17.846437 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:43:17.846412    5428 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-127.ec2.internal"
Oct 29 22:43:17.883184 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883157    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 29 22:43:17.883332 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883283    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x4fl8\""
Oct 29 22:43:17.883332 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883296    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 29 22:43:17.883440 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883429    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 29 22:43:17.883492 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883458    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n7jjw\""
Oct 29 22:43:17.883492 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883474    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d6hqb\""
Oct 29 22:43:17.883614 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883493    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 22:43:17.883614 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883463    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 29 22:43:17.883614 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883542    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 22:43:17.883614 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883571    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 29 22:43:17.883614 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883600    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 29 22:43:17.883614 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883615    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 29 22:43:17.883917 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883627    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dxz6f\""
Oct 29 22:43:17.883917 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883634    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 29 22:43:17.883917 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883649    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dk7bsml1jhsna\""
Oct 29 22:43:17.883917 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883677    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gx7tn\""
Oct 29 22:43:17.883917 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883683    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 29 22:43:17.883917 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883690    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 29 22:43:17.883917 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883679    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 29 22:43:17.883917 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883830    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 29 22:43:17.883917 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883874    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 29 22:43:17.883917 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.883905    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 29 22:43:17.884236 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.884225    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tlsz7\""
Oct 29 22:43:17.884825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.884792    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 29 22:43:17.885062 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.885041    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 29 22:43:17.885139 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.885052    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 29 22:43:17.885139 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.885132    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 29 22:43:17.885237 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.885146    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 29 22:43:17.885237 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.885151    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 29 22:43:17.885237 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.885183    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 29 22:43:17.885237 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.885188    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 29 22:43:17.885411 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.885256    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 22:43:17.885411 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.885267    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 29 22:43:17.885411 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.885302    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 22:43:17.885411 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.885381    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 29 22:43:17.887403 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.887382    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.887472 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.887411    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-config\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:17.887472 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.887433    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-socket-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.887472 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.887457    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d6fdbfe-0357-4139-a034-28faec681123-multus-daemon-config\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.887578 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.887483    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q96hr\" (UniqueName: \"kubernetes.io/projected/75ac2c49-e052-4613-a573-0a8cf1475ad3-kube-api-access-q96hr\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:43:17.887578 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.887539    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca99b14b-c183-4966-85c2-b685be64e4cb-tmp-dir\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:43:17.887638 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.887618    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca99b14b-c183-4966-85c2-b685be64e4cb-tmp-dir\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:43:17.887739 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.887722    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-config\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:17.887838 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.887822    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d6fdbfe-0357-4139-a034-28faec681123-multus-daemon-config\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.887880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.887866    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-sys-fs\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.887915 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.887883    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.887915 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.887907    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-lib-modules\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.888030 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888015    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.888065 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888046    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.888098 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888073    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.888313 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888295    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmwh\" (UniqueName: \"kubernetes.io/projected/d7f074e8-1737-4f5c-99cd-928d9c565c17-kube-api-access-6hmwh\") pod \"network-metrics-daemon-prbcb\" (UID: \"d7f074e8-1737-4f5c-99cd-928d9c565c17\") " pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:43:17.888351 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888314    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.888351 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888328    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/98c4af2a-2166-4fa0-99d0-662be28b6899-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:43:17.888414 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888357    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-metrics-client-ca\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.888414 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888382    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-var-lib-kubelet\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.888414 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888402    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.888530 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888407    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.888567 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888553    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-wtmp\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.888598 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888588    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fea00aae-5dd9-48ef-b7d4-af08eed1481d-konnectivity-ca\") pod \"konnectivity-agent-5jn2n\" (UID: \"fea00aae-5dd9-48ef-b7d4-af08eed1481d\") " pod="kube-system/konnectivity-agent-5jn2n"
Oct 29 22:43:17.888626 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888592    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.888626 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888602    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/98c4af2a-2166-4fa0-99d0-662be28b6899-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:43:17.888626 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888614    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e50192cd-cb7b-4b3e-978f-661093207b81-hosts-file\") pod \"node-resolver-q8pg5\" (UID: \"e50192cd-cb7b-4b3e-978f-661093207b81\") " pod="openshift-dns/node-resolver-q8pg5"
Oct 29 22:43:17.888710 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888668    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-sysctl-d\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.888782 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888766    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-root\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.888836 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888808    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-959fd\" (UniqueName: \"kubernetes.io/projected/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-kube-api-access-959fd\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.888836 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888794    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fea00aae-5dd9-48ef-b7d4-af08eed1481d-konnectivity-ca\") pod \"konnectivity-agent-5jn2n\" (UID: \"fea00aae-5dd9-48ef-b7d4-af08eed1481d\") " pod="kube-system/konnectivity-agent-5jn2n"
Oct 29 22:43:17.888935 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888849    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-oauth-config\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:17.888935 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888875    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-metrics-client-ca\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.888935 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888878    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grckl\" (UniqueName: \"kubernetes.io/projected/d31c1857-282f-4a04-89a9-883b1c2a7c19-kube-api-access-grckl\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:17.888935 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888915    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs\") pod \"network-metrics-daemon-prbcb\" (UID: \"d7f074e8-1737-4f5c-99cd-928d9c565c17\") " pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:43:17.889167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888937    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g26rs\" (UniqueName: \"kubernetes.io/projected/ca99b14b-c183-4966-85c2-b685be64e4cb-kube-api-access-g26rs\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:43:17.889167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888954    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-trusted-ca-bundle\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:17.889167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.888974    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-oauth-serving-cert\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:17.889167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889039    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-oauth-config\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:17.889167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889046    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6q45\" (UniqueName: \"kubernetes.io/projected/765947ed-ccf0-4296-aac8-eace5ce6de20-kube-api-access-p6q45\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.889167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889077    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l295\" (UniqueName: \"kubernetes.io/projected/0d1b5f20-523b-4c79-a942-b0dfce8f64f9-kube-api-access-4l295\") pod \"iptables-alerter-2q4dp\" (UID: \"0d1b5f20-523b-4c79-a942-b0dfce8f64f9\") " pod="openshift-network-operator/iptables-alerter-2q4dp"
Oct 29 22:43:17.889167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889103    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-hostroot\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.889167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889128    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-serving-cert\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:17.889167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889154    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86e95d56-5a28-4cdd-b23d-e56a55de786f-cert\") pod \"ingress-canary-dgv4b\" (UID: \"86e95d56-5a28-4cdd-b23d-e56a55de786f\") " pod="openshift-ingress-canary/ingress-canary-dgv4b"
Oct 29 22:43:17.889167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889160    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f074e8-1737-4f5c-99cd-928d9c565c17-metrics-certs\") pod \"network-metrics-daemon-prbcb\" (UID: \"d7f074e8-1737-4f5c-99cd-928d9c565c17\") " pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:43:17.889705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889196    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbs5z\" (UniqueName: \"kubernetes.io/projected/86e95d56-5a28-4cdd-b23d-e56a55de786f-kube-api-access-qbs5z\") pod \"ingress-canary-dgv4b\" (UID: \"86e95d56-5a28-4cdd-b23d-e56a55de786f\") " pod="openshift-ingress-canary/ingress-canary-dgv4b"
Oct 29 22:43:17.889705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889229    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-systemd-units\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.889705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889253    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-run-ovn\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.889705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889275    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-node-log\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.889705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889283    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-oauth-serving-cert\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:17.889705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889397    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86e95d56-5a28-4cdd-b23d-e56a55de786f-cert\") pod \"ingress-canary-dgv4b\" (UID: \"86e95d56-5a28-4cdd-b23d-e56a55de786f\") " pod="openshift-ingress-canary/ingress-canary-dgv4b"
Oct 29 22:43:17.889705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889432    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-serving-cert\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:17.889705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889503    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.889705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889545    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75ac2c49-e052-4613-a573-0a8cf1475ad3-data-volume\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:43:17.889705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889574    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-trusted-ca-bundle\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:17.889705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889603    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-service-ca\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:17.889705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889632    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a381148-a1d6-404f-94ca-eed5b9481b7a-tmp\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:17.889705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889655    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-trusted-ca-bundle\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:17.889705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889680    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-var-lib-cni-multus\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.889705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889703    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75ac2c49-e052-4613-a573-0a8cf1475ad3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:43:17.890192 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889734    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.890192 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889757    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-oauth-serving-cert\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:17.890192 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889780    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98c4af2a-2166-4fa0-99d0-662be28b6899-system-cni-dir\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:43:17.890192 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889827    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-registration-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.890192 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889841    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75ac2c49-e052-4613-a573-0a8cf1475ad3-data-volume\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:43:17.890192 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.889975    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-service-ca\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:17.890192 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890023    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75ac2c49-e052-4613-a573-0a8cf1475ad3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:43:17.890192 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890087    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-oauth-serving-cert\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:17.890192 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890108    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d6fdbfe-0357-4139-a034-28faec681123-cni-binary-copy\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.890192 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890152    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.890192 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890155    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a381148-a1d6-404f-94ca-eed5b9481b7a-tmp\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:17.890192 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890170    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-oauth-config\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:17.890695 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890276    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtrgr\" (UniqueName: \"kubernetes.io/projected/5a381148-a1d6-404f-94ca-eed5b9481b7a-kube-api-access-gtrgr\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:17.890695 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890304    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d6fdbfe-0357-4139-a034-28faec681123-cni-binary-copy\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.890695 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890311    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.890695 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890416    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f8gp\" (UniqueName: \"kubernetes.io/projected/da8029d1-50de-4612-adce-3a3f582c74d8-kube-api-access-9f8gp\") pod \"node-ca-n27f7\" (UID: \"da8029d1-50de-4612-adce-3a3f582c74d8\") " pod="openshift-image-registry/node-ca-n27f7"
Oct 29 22:43:17.890695 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890475    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-oauth-config\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:17.890695 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890503    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.890695 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890561    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.890695 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890603    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-service-ca\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:17.890695 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890698    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-env-overrides\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.891168 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890723    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-run-k8s-cni-cncf-io\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.891168 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890729    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.891168 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890890    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-service-ca\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:17.891168 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890917    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-env-overrides\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.891168 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.890918    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/38dfb740-4469-4101-a309-4070aeb2dfe3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:43:17.891168 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891125    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/38dfb740-4469-4101-a309-4070aeb2dfe3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:43:17.891168 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891158    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-cni-bin\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.891504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891189    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-run-multus-certs\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.891504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891215    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-sysctl-conf\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.891504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891249    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38dfb740-4469-4101-a309-4070aeb2dfe3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:43:17.891504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891272    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-slash\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.891504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891295    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-sysconfig\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.891504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891315    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-run\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.891504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891361    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-host\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.891504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891387    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.891504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891411    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-etc-selinux\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.891504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891435    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-multus-conf-dir\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.891504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891458    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-run-systemd\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.891504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891481    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fea00aae-5dd9-48ef-b7d4-af08eed1481d-agent-certs\") pod \"konnectivity-agent-5jn2n\" (UID: \"fea00aae-5dd9-48ef-b7d4-af08eed1481d\") " pod="kube-system/konnectivity-agent-5jn2n"
Oct 29 22:43:17.891504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891503    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-system-cni-dir\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891526    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-cnibin\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891605    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-run-netns\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891634    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ndcg\" (UniqueName: \"kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg\") pod \"network-check-target-lqjl6\" (UID: \"8ba702dc-08af-46e3-8235-61da0c303efd\") " pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891662    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0d1b5f20-523b-4c79-a942-b0dfce8f64f9-iptables-alerter-script\") pod \"iptables-alerter-2q4dp\" (UID: \"0d1b5f20-523b-4c79-a942-b0dfce8f64f9\") " pod="openshift-network-operator/iptables-alerter-2q4dp"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891686    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-tmp\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891707    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40fb2dc7-336c-4483-93e7-68309e345b22-config-out\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891730    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-multus-socket-dir-parent\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891754    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75ac2c49-e052-4613-a573-0a8cf1475ad3-crio-socket\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891779    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891822    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891850    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-var-lib-kubelet\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891872    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-run-netns\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891899    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-etc-openvswitch\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891925    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-ovn-node-metrics-cert\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891948    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d1b5f20-523b-4c79-a942-b0dfce8f64f9-host-slash\") pod \"iptables-alerter-2q4dp\" (UID: \"0d1b5f20-523b-4c79-a942-b0dfce8f64f9\") " pod="openshift-network-operator/iptables-alerter-2q4dp"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891971    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-var-lib-cni-bin\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.891995    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-tls\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892011    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38dfb740-4469-4101-a309-4070aeb2dfe3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:43:17.892074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892031    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98c4af2a-2166-4fa0-99d0-662be28b6899-cni-binary-copy\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892202    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892231    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-kubernetes\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892258    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-run-openvswitch\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892268    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fea00aae-5dd9-48ef-b7d4-af08eed1481d-agent-certs\") pod \"konnectivity-agent-5jn2n\" (UID: \"fea00aae-5dd9-48ef-b7d4-af08eed1481d\") " pod="kube-system/konnectivity-agent-5jn2n"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892294    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c68dp\" (UniqueName: \"kubernetes.io/projected/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-kube-api-access-c68dp\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892322    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-textfile\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892381    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/98c4af2a-2166-4fa0-99d0-662be28b6899-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892406    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-serving-cert\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892430    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-config\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892455    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892489    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892624    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0d1b5f20-523b-4c79-a942-b0dfce8f64f9-iptables-alerter-script\") pod \"iptables-alerter-2q4dp\" (UID: \"0d1b5f20-523b-4c79-a942-b0dfce8f64f9\") " pod="openshift-network-operator/iptables-alerter-2q4dp"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892699    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50192cd-cb7b-4b3e-978f-661093207b81-tmp-dir\") pod \"node-resolver-q8pg5\" (UID: \"e50192cd-cb7b-4b3e-978f-661093207b81\") " pod="openshift-dns/node-resolver-q8pg5"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892724    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-sys\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892751    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qckd7\" (UniqueName: \"kubernetes.io/projected/752e574a-9cd1-40a1-a65e-7b2aabd4434e-kube-api-access-qckd7\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892780    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892826    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm689\" (UniqueName: \"kubernetes.io/projected/40fb2dc7-336c-4483-93e7-68309e345b22-kube-api-access-mm689\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892849    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98c4af2a-2166-4fa0-99d0-662be28b6899-cni-binary-copy\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:43:17.892850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892858    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6edb8e5f-e185-48be-8fed-5ac6899134fa-dbus\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892881    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98c4af2a-2166-4fa0-99d0-662be28b6899-cnibin\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892905    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98c4af2a-2166-4fa0-99d0-662be28b6899-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892927    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-log-socket\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892941    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-textfile\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892951    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-modprobe-d\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.892977    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-config\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893000    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-serving-cert\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893038    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-systemd\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893071    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-oauth-config\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893095    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-oauth-serving-cert\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893110    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/98c4af2a-2166-4fa0-99d0-662be28b6899-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893204    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl249\" (UniqueName: \"kubernetes.io/projected/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-kube-api-access-wl249\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893230    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6edb8e5f-e185-48be-8fed-5ac6899134fa-kubelet-config\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893255    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-var-lib-openvswitch\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893278    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-service-ca\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893302    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-cni-netd\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893324    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da8029d1-50de-4612-adce-3a3f582c74d8-host\") pod \"node-ca-n27f7\" (UID: \"da8029d1-50de-4612-adce-3a3f582c74d8\") " pod="openshift-image-registry/node-ca-n27f7"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893347    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da8029d1-50de-4612-adce-3a3f582c74d8-serviceca\") pod \"node-ca-n27f7\" (UID: \"da8029d1-50de-4612-adce-3a3f582c74d8\") " pod="openshift-image-registry/node-ca-n27f7"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893371    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srpcs\" (UniqueName: \"kubernetes.io/projected/7d6fdbfe-0357-4139-a034-28faec681123-kube-api-access-srpcs\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893396    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-web-config\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893420    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38dfb740-4469-4101-a309-4070aeb2dfe3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893447    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893475    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893499    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-ovnkube-script-lib\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893524    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6ncm\" (UniqueName: \"kubernetes.io/projected/e50192cd-cb7b-4b3e-978f-661093207b81-kube-api-access-l6ncm\") pod \"node-resolver-q8pg5\" (UID: \"e50192cd-cb7b-4b3e-978f-661093207b81\") " pod="openshift-dns/node-resolver-q8pg5"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893550    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-multus-cni-dir\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893587    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-tuned\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893614    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/752e574a-9cd1-40a1-a65e-7b2aabd4434e-tmp\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.893635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893638    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-etc-kubernetes\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893664    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75ac2c49-e052-4613-a573-0a8cf1475ad3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893701    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwgf\" (UniqueName: \"kubernetes.io/projected/38dfb740-4469-4101-a309-4070aeb2dfe3-kube-api-access-thwgf\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893727    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40fb2dc7-336c-4483-93e7-68309e345b22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893948    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.893977    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.894003    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.894029    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d31c1857-282f-4a04-89a9-883b1c2a7c19-tmp\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.894056    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb7k9\" (UniqueName: \"kubernetes.io/projected/98c4af2a-2166-4fa0-99d0-662be28b6899-kube-api-access-gb7k9\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.894083    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-kubelet\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.894110    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-ovnkube-config\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.894135    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-device-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.894160    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ca99b14b-c183-4966-85c2-b685be64e4cb-config-volume\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.894183    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-sys\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.894209    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.894242    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca99b14b-c183-4966-85c2-b685be64e4cb-metrics-tls\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.894244    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-config\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.894269    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-config\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.894407    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40fb2dc7-336c-4483-93e7-68309e345b22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.894578    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-oauth-serving-cert\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.894781    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.894891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.894894    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-config\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:17.895771 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.894948    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.895771 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.895056    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-service-ca\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:17.895771 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.895260    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-ovn-node-metrics-cert\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.895771 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.895460    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-tls\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.895771 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.895544    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da8029d1-50de-4612-adce-3a3f582c74d8-serviceca\") pod \"node-ca-n27f7\" (UID: \"da8029d1-50de-4612-adce-3a3f582c74d8\") " pod="openshift-image-registry/node-ca-n27f7"
Oct 29 22:43:17.895771 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.895645    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-serving-cert\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:17.896072 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.895921    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-config\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.896072 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.895933    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-web-config\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.896159 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.896089    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38dfb740-4469-4101-a309-4070aeb2dfe3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:43:17.896159 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.896139    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.896243 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.896202    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-tmp\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:17.896289 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.896245    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40fb2dc7-336c-4483-93e7-68309e345b22-config-out\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.896341 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.896300    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50192cd-cb7b-4b3e-978f-661093207b81-tmp-dir\") pod \"node-resolver-q8pg5\" (UID: \"e50192cd-cb7b-4b3e-978f-661093207b81\") " pod="openshift-dns/node-resolver-q8pg5"
Oct 29 22:43:17.896390 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.896365    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6edb8e5f-e185-48be-8fed-5ac6899134fa-original-pull-secret\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:43:17.896571 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.896552    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-ovnkube-script-lib\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.896668 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.896648    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.896823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.896786    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-tuned\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.896901 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.896877    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-serving-cert\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:17.896947 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.896933    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/752e574a-9cd1-40a1-a65e-7b2aabd4434e-tmp\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.897042 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.897026    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-oauth-config\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:17.897152 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.897135    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75ac2c49-e052-4613-a573-0a8cf1475ad3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:43:17.897536 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.897517    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.897680 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.897660    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.897741 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.897728    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d31c1857-282f-4a04-89a9-883b1c2a7c19-tmp\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:17.897792 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.897778    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40fb2dc7-336c-4483-93e7-68309e345b22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.897954 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.897933    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-ovnkube-config\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.898011 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.897949    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ca99b14b-c183-4966-85c2-b685be64e4cb-config-volume\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:43:17.898163 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.898147    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca99b14b-c183-4966-85c2-b685be64e4cb-metrics-tls\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:43:17.900560 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.900480    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 29 22:43:17.900560 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.900520    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 29 22:43:17.900560 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.900526    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 29 22:43:17.900825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.900786    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-trusted-ca-bundle\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:17.915618 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.904054    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.915618 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.911312    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q96hr\" (UniqueName: \"kubernetes.io/projected/75ac2c49-e052-4613-a573-0a8cf1475ad3-kube-api-access-q96hr\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:43:17.915618 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.914364    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f8gp\" (UniqueName: \"kubernetes.io/projected/da8029d1-50de-4612-adce-3a3f582c74d8-kube-api-access-9f8gp\") pod \"node-ca-n27f7\" (UID: \"da8029d1-50de-4612-adce-3a3f582c74d8\") " pod="openshift-image-registry/node-ca-n27f7"
Oct 29 22:43:17.916301 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.916277    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-trusted-ca-bundle\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:17.916574 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.916554    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-trusted-ca-bundle\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:17.921633 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.921287    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbs5z\" (UniqueName: \"kubernetes.io/projected/86e95d56-5a28-4cdd-b23d-e56a55de786f-kube-api-access-qbs5z\") pod \"ingress-canary-dgv4b\" (UID: \"86e95d56-5a28-4cdd-b23d-e56a55de786f\") " pod="openshift-ingress-canary/ingress-canary-dgv4b"
Oct 29 22:43:17.921633 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.921359    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g26rs\" (UniqueName: \"kubernetes.io/projected/ca99b14b-c183-4966-85c2-b685be64e4cb-kube-api-access-g26rs\") pod \"dns-default-dcx8f\" (UID: \"ca99b14b-c183-4966-85c2-b685be64e4cb\") " pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:43:17.921633 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.921598    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ndcg\" (UniqueName: \"kubernetes.io/projected/8ba702dc-08af-46e3-8235-61da0c303efd-kube-api-access-5ndcg\") pod \"network-check-target-lqjl6\" (UID: \"8ba702dc-08af-46e3-8235-61da0c303efd\") " pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:43:17.921844 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.921643    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtrgr\" (UniqueName: \"kubernetes.io/projected/5a381148-a1d6-404f-94ca-eed5b9481b7a-kube-api-access-gtrgr\") pod \"console-6bff8847db-vg5bs\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") " pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:17.922371 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.922347    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grckl\" (UniqueName: \"kubernetes.io/projected/d31c1857-282f-4a04-89a9-883b1c2a7c19-kube-api-access-grckl\") pod \"console-774f56585c-2x9nc\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") " pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:17.923265 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.923116    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c68dp\" (UniqueName: \"kubernetes.io/projected/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-kube-api-access-c68dp\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.923265 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.923230    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l295\" (UniqueName: \"kubernetes.io/projected/0d1b5f20-523b-4c79-a942-b0dfce8f64f9-kube-api-access-4l295\") pod \"iptables-alerter-2q4dp\" (UID: \"0d1b5f20-523b-4c79-a942-b0dfce8f64f9\") " pod="openshift-network-operator/iptables-alerter-2q4dp"
Oct 29 22:43:17.923265 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.923258    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmwh\" (UniqueName: \"kubernetes.io/projected/d7f074e8-1737-4f5c-99cd-928d9c565c17-kube-api-access-6hmwh\") pod \"network-metrics-daemon-prbcb\" (UID: \"d7f074e8-1737-4f5c-99cd-928d9c565c17\") " pod="openshift-multus/network-metrics-daemon-prbcb"
Oct 29 22:43:17.923782 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.923758    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-959fd\" (UniqueName: \"kubernetes.io/projected/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-kube-api-access-959fd\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.923913 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.923856    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb7k9\" (UniqueName: \"kubernetes.io/projected/98c4af2a-2166-4fa0-99d0-662be28b6899-kube-api-access-gb7k9\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:43:17.923991 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.923972    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6q45\" (UniqueName: \"kubernetes.io/projected/765947ed-ccf0-4296-aac8-eace5ce6de20-kube-api-access-p6q45\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.924998 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.924974    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm689\" (UniqueName: \"kubernetes.io/projected/40fb2dc7-336c-4483-93e7-68309e345b22-kube-api-access-mm689\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.927251 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.926899    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6ncm\" (UniqueName: \"kubernetes.io/projected/e50192cd-cb7b-4b3e-978f-661093207b81-kube-api-access-l6ncm\") pod \"node-resolver-q8pg5\" (UID: \"e50192cd-cb7b-4b3e-978f-661093207b81\") " pod="openshift-dns/node-resolver-q8pg5"
Oct 29 22:43:17.927251 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.927154    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srpcs\" (UniqueName: \"kubernetes.io/projected/7d6fdbfe-0357-4139-a034-28faec681123-kube-api-access-srpcs\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.927251 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.927206    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl249\" (UniqueName: \"kubernetes.io/projected/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-kube-api-access-wl249\") pod \"console-56797698b7-xlsbd\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") " pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:17.927654 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.927635    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thwgf\" (UniqueName: \"kubernetes.io/projected/38dfb740-4469-4101-a309-4070aeb2dfe3-kube-api-access-thwgf\") pod \"prometheus-operator-9cd89b74c-87hkr\" (UID: \"38dfb740-4469-4101-a309-4070aeb2dfe3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-87hkr"
Oct 29 22:43:17.927710 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.927635    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qckd7\" (UniqueName: \"kubernetes.io/projected/752e574a-9cd1-40a1-a65e-7b2aabd4434e-kube-api-access-qckd7\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.930068 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.930042    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40fb2dc7-336c-4483-93e7-68309e345b22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40fb2dc7-336c-4483-93e7-68309e345b22\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:17.995296 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995216    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-sys\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.995296 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995250    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6edb8e5f-e185-48be-8fed-5ac6899134fa-dbus\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:43:17.995296 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995267    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98c4af2a-2166-4fa0-99d0-662be28b6899-cnibin\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:43:17.995544 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995328    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6edb8e5f-e185-48be-8fed-5ac6899134fa-dbus\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:43:17.995544 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995330    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-sys\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.995544 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995369    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98c4af2a-2166-4fa0-99d0-662be28b6899-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:43:17.995544 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995400    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-log-socket\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.995544 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995409    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98c4af2a-2166-4fa0-99d0-662be28b6899-cnibin\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:43:17.995544 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995417    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-modprobe-d\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.995544 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995452    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-log-socket\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.995544 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995458    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98c4af2a-2166-4fa0-99d0-662be28b6899-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:43:17.995544 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995473    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-systemd\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.995544 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995499    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-modprobe-d\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.995544 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995505    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6edb8e5f-e185-48be-8fed-5ac6899134fa-kubelet-config\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:43:17.995544 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995537    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-var-lib-openvswitch\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.995544 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995550    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-systemd\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995560    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-cni-netd\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995575    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-var-lib-openvswitch\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995586    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da8029d1-50de-4612-adce-3a3f582c74d8-host\") pod \"node-ca-n27f7\" (UID: \"da8029d1-50de-4612-adce-3a3f582c74d8\") " pod="openshift-image-registry/node-ca-n27f7"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995620    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995563    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6edb8e5f-e185-48be-8fed-5ac6899134fa-kubelet-config\") pod \"global-pull-secret-syncer-qmfpg\" (UID: \"6edb8e5f-e185-48be-8fed-5ac6899134fa\") " pod="kube-system/global-pull-secret-syncer-qmfpg"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995621    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-cni-netd\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995648    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-multus-cni-dir\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995656    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da8029d1-50de-4612-adce-3a3f582c74d8-host\") pod \"node-ca-n27f7\" (UID: \"da8029d1-50de-4612-adce-3a3f582c74d8\") " pod="openshift-image-registry/node-ca-n27f7"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995673    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-etc-kubernetes\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995681    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-multus-cni-dir\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995673    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995699    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-etc-kubernetes\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995712    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-kubelet\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995754    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-kubelet\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995764    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-device-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995783    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-sys\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995811    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995832    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-device-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995846    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-sys\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.996063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.995913    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996226    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-socket-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996245    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-sys-fs\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996261    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-lib-modules\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996280    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-var-lib-kubelet\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996296    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-wtmp\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996311    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e50192cd-cb7b-4b3e-978f-661093207b81-hosts-file\") pod \"node-resolver-q8pg5\" (UID: \"e50192cd-cb7b-4b3e-978f-661093207b81\") " pod="openshift-dns/node-resolver-q8pg5"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996320    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-sys-fs\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996329    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-sysctl-d\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996333    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-var-lib-kubelet\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996350    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-root\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996365    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e50192cd-cb7b-4b3e-978f-661093207b81-hosts-file\") pod \"node-resolver-q8pg5\" (UID: \"e50192cd-cb7b-4b3e-978f-661093207b81\") " pod="openshift-dns/node-resolver-q8pg5"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996373    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-node-exporter-wtmp\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996381    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-hostroot\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996411    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c44b4b5f-8aa2-4cf2-b72a-16b5002260b6-root\") pod \"node-exporter-5pzlf\" (UID: \"c44b4b5f-8aa2-4cf2-b72a-16b5002260b6\") " pod="openshift-monitoring/node-exporter-5pzlf"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996416    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-systemd-units\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996422    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-hostroot\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996357    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-socket-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996441    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-run-ovn\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996434    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-sysctl-d\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996456    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-lib-modules\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996468    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-node-log\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996460    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-systemd-units\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996497    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996516    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-node-log\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996536    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-var-lib-cni-multus\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996546    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996554    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98c4af2a-2166-4fa0-99d0-662be28b6899-system-cni-dir\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996501    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-run-ovn\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996581    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98c4af2a-2166-4fa0-99d0-662be28b6899-system-cni-dir\") pod \"multus-additional-cni-plugins-mlsbk\" (UID: \"98c4af2a-2166-4fa0-99d0-662be28b6899\") " pod="openshift-multus/multus-additional-cni-plugins-mlsbk"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996585    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-registration-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996613    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-var-lib-cni-multus\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996620    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-run-k8s-cni-cncf-io\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996647    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-cni-bin\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996672    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-run-multus-certs\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996675    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-run-k8s-cni-cncf-io\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996678    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-registration-dir\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996696    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-sysctl-conf\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.996768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.996712    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-cni-bin\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997085    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-slash\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997101    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-run-multus-certs\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997133    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-sysconfig\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997156    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-sysctl-conf\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997190    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-slash\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997218    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-sysconfig\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997232    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-run\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997258    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-host\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997291    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-etc-selinux\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997346    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-run\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997349    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-host\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997411    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-multus-conf-dir\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997413    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/765947ed-ccf0-4296-aac8-eace5ce6de20-etc-selinux\") pod \"aws-ebs-csi-driver-node-wq666\" (UID: \"765947ed-ccf0-4296-aac8-eace5ce6de20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wq666"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997452    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-multus-conf-dir\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997484    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-run-systemd\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997513    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-system-cni-dir\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997536    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-cnibin\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997562    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-run-netns\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997594    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-multus-socket-dir-parent\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997626    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75ac2c49-e052-4613-a573-0a8cf1475ad3-crio-socket\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997656    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-var-lib-kubelet\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997686    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-run-netns\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997713    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-etc-openvswitch\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997742    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d1b5f20-523b-4c79-a942-b0dfce8f64f9-host-slash\") pod \"iptables-alerter-2q4dp\" (UID: \"0d1b5f20-523b-4c79-a942-b0dfce8f64f9\") " pod="openshift-network-operator/iptables-alerter-2q4dp"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997772    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-var-lib-cni-bin\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997814    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-kubernetes\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997842    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-run-openvswitch\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997946    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-run-openvswitch\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.997994    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-run-systemd\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.998047    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-system-cni-dir\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.998125    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-run-netns\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.998185    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-multus-socket-dir-parent\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.998243    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75ac2c49-e052-4613-a573-0a8cf1475ad3-crio-socket\") pod \"insights-runtime-extractor-gpthh\" (UID: \"75ac2c49-e052-4613-a573-0a8cf1475ad3\") " pod="openshift-insights/insights-runtime-extractor-gpthh"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.998293    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-var-lib-kubelet\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.998346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.998339    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-host-run-netns\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.999771 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.998382    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/501b61a4-5a18-4c9f-8b37-0b162c9c2c2b-etc-openvswitch\") pod \"ovnkube-node-4xc2d\" (UID: \"501b61a4-5a18-4c9f-8b37-0b162c9c2c2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:17.999771 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.998429    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d1b5f20-523b-4c79-a942-b0dfce8f64f9-host-slash\") pod \"iptables-alerter-2q4dp\" (UID: \"0d1b5f20-523b-4c79-a942-b0dfce8f64f9\") " pod="openshift-network-operator/iptables-alerter-2q4dp"
Oct 29 22:43:17.999771 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.998472    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-host-var-lib-cni-bin\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:17.999771 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.998524    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/752e574a-9cd1-40a1-a65e-7b2aabd4434e-etc-kubernetes\") pod \"tuned-4bgs7\" (UID: \"752e574a-9cd1-40a1-a65e-7b2aabd4434e\") " pod="openshift-cluster-node-tuning-operator/tuned-4bgs7"
Oct 29 22:43:17.999771 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:17.998599    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d6fdbfe-0357-4139-a034-28faec681123-cnibin\") pod \"multus-8dtlv\" (UID: \"7d6fdbfe-0357-4139-a034-28faec681123\") " pod="openshift-multus/multus-8dtlv"
Oct 29 22:43:18.355064 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:18.354983    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dcx8f"
Oct 29 22:43:19.119531 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:19.119498    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:19.133506 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:19.133476    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:20.255333 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:20.255302    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:20.341492 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:20.341435    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:20.454045 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:20.454015    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:21.864524 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:21.864491    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:22.206279 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:22.206173    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:22.326935 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:22.326897    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:22.609091 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:22.609005    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:22.793590 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:22.793553    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:22.850731 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:22.850703    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:43:23.249728 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:23.249703    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:23.250188 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:23.249914    5428 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:43:27.206209 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:27.206183    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5jn2n"
Oct 29 22:43:27.460634 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:27.460559    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:43:27.715159 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:27.715061    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xc2d"
Oct 29 22:43:28.029870 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:28.029771    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:28.590226 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:28.590195    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:29.525044 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:29.524990    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:29.642364 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:29.642329    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:31.869760 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:31.869732    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:31.873436 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:31.873417    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:43:32.224848 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:32.224757    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:32.333888 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:32.333858    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:39.864137 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:39.864100    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:39.988255 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:39.988221    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:42.800782 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:42.800715    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6bff8847db-vg5bs" podUID="5a381148-a1d6-404f-94ca-eed5b9481b7a" containerName="console" containerID="cri-o://08a02334529f0003ff8281388c584112623150bfc959c4a7ac12df4773c37929" gracePeriod=15
Oct 29 22:43:42.804751 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:42.804717    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-56797698b7-xlsbd" podUID="9e8347f7-98bc-4826-a26d-2ad74ec8b27e" containerName="console" containerID="cri-o://acec23f1e3c79e2cfcf49ca0b451890c1b403af2ebe102a46cb2dcf9c835c289" gracePeriod=15
Oct 29 22:43:43.086602 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.086580    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:43.089408 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.089390    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:43.111667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.111638    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:43.191910 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.191878    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a381148-a1d6-404f-94ca-eed5b9481b7a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a381148-a1d6-404f-94ca-eed5b9481b7a-tmp\") pod \"5a381148-a1d6-404f-94ca-eed5b9481b7a\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") "
Oct 29 22:43:43.191910 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.191917    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-tmp\") pod \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") "
Oct 29 22:43:43.192137 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.191942    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a381148-a1d6-404f-94ca-eed5b9481b7a-kube-api-access-gtrgr\" (UniqueName: \"kubernetes.io/projected/5a381148-a1d6-404f-94ca-eed5b9481b7a-kube-api-access-gtrgr\") pod \"5a381148-a1d6-404f-94ca-eed5b9481b7a\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") "
Oct 29 22:43:43.192137 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.191961    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-trusted-ca-bundle\") pod \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") "
Oct 29 22:43:43.192137 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.191978    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-serving-cert\") pod \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") "
Oct 29 22:43:43.192137 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192012    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-oauth-config\") pod \"5a381148-a1d6-404f-94ca-eed5b9481b7a\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") "
Oct 29 22:43:43.192137 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192041    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-oauth-serving-cert\") pod \"5a381148-a1d6-404f-94ca-eed5b9481b7a\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") "
Oct 29 22:43:43.192137 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192072    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-config\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-config\") pod \"5a381148-a1d6-404f-94ca-eed5b9481b7a\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") "
Oct 29 22:43:43.192137 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192106    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-trusted-ca-bundle\") pod \"5a381148-a1d6-404f-94ca-eed5b9481b7a\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") "
Oct 29 22:43:43.192137 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192134    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-serving-cert\") pod \"5a381148-a1d6-404f-94ca-eed5b9481b7a\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") "
Oct 29 22:43:43.192517 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192161    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-config\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-config\") pod \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") "
Oct 29 22:43:43.192517 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192199    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-oauth-config\") pod \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") "
Oct 29 22:43:43.192517 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192246    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a381148-a1d6-404f-94ca-eed5b9481b7a-tmp" pod "5a381148-a1d6-404f-94ca-eed5b9481b7a" (UID: "5a381148-a1d6-404f-94ca-eed5b9481b7a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:43.192517 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192261    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-tmp" pod "9e8347f7-98bc-4826-a26d-2ad74ec8b27e" (UID: "9e8347f7-98bc-4826-a26d-2ad74ec8b27e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:43:43.192517 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192257    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-oauth-serving-cert\") pod \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") "
Oct 29 22:43:43.192517 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192335    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-service-ca\") pod \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") "
Oct 29 22:43:43.192517 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192390    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-kube-api-access-wl249\" (UniqueName: \"kubernetes.io/projected/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-kube-api-access-wl249\") pod \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\" (UID: \"9e8347f7-98bc-4826-a26d-2ad74ec8b27e\") "
Oct 29 22:43:43.192517 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192422    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-service-ca\") pod \"5a381148-a1d6-404f-94ca-eed5b9481b7a\" (UID: \"5a381148-a1d6-404f-94ca-eed5b9481b7a\") "
Oct 29 22:43:43.192517 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192416    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-trusted-ca-bundle" pod "9e8347f7-98bc-4826-a26d-2ad74ec8b27e" (UID: "9e8347f7-98bc-4826-a26d-2ad74ec8b27e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:43:43.192999 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192660    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-oauth-serving-cert" pod "9e8347f7-98bc-4826-a26d-2ad74ec8b27e" (UID: "9e8347f7-98bc-4826-a26d-2ad74ec8b27e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:43:43.192999 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192689    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-config" pod "9e8347f7-98bc-4826-a26d-2ad74ec8b27e" (UID: "9e8347f7-98bc-4826-a26d-2ad74ec8b27e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:43:43.192999 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192707    5428 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a381148-a1d6-404f-94ca-eed5b9481b7a-tmp\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.192999 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192728    5428 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-tmp\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.192999 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192743    5428 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-trusted-ca-bundle\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.192999 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.192760    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-trusted-ca-bundle" pod "5a381148-a1d6-404f-94ca-eed5b9481b7a" (UID: "5a381148-a1d6-404f-94ca-eed5b9481b7a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:43:43.193344 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.193136    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-oauth-serving-cert" pod "5a381148-a1d6-404f-94ca-eed5b9481b7a" (UID: "5a381148-a1d6-404f-94ca-eed5b9481b7a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:43:43.193344 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.193151    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-service-ca" pod "9e8347f7-98bc-4826-a26d-2ad74ec8b27e" (UID: "9e8347f7-98bc-4826-a26d-2ad74ec8b27e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:43:43.193344 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.193185    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-config" pod "5a381148-a1d6-404f-94ca-eed5b9481b7a" (UID: "5a381148-a1d6-404f-94ca-eed5b9481b7a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:43:43.195357 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.193364    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-service-ca" pod "5a381148-a1d6-404f-94ca-eed5b9481b7a" (UID: "5a381148-a1d6-404f-94ca-eed5b9481b7a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:43:43.197157 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.197130    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-serving-cert" pod "5a381148-a1d6-404f-94ca-eed5b9481b7a" (UID: "5a381148-a1d6-404f-94ca-eed5b9481b7a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:43:43.197278 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.197150    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a381148-a1d6-404f-94ca-eed5b9481b7a-kube-api-access-gtrgr" pod "5a381148-a1d6-404f-94ca-eed5b9481b7a" (UID: "5a381148-a1d6-404f-94ca-eed5b9481b7a"). InnerVolumeSpecName "kube-api-access-gtrgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:43:43.197278 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.197134    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-oauth-config" pod "9e8347f7-98bc-4826-a26d-2ad74ec8b27e" (UID: "9e8347f7-98bc-4826-a26d-2ad74ec8b27e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:43:43.197507 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.197479    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-serving-cert" pod "9e8347f7-98bc-4826-a26d-2ad74ec8b27e" (UID: "9e8347f7-98bc-4826-a26d-2ad74ec8b27e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:43:43.197661 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.197645    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-kube-api-access-wl249" pod "9e8347f7-98bc-4826-a26d-2ad74ec8b27e" (UID: "9e8347f7-98bc-4826-a26d-2ad74ec8b27e"). InnerVolumeSpecName "kube-api-access-wl249". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:43:43.197977 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.197955    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-oauth-config" pod "5a381148-a1d6-404f-94ca-eed5b9481b7a" (UID: "5a381148-a1d6-404f-94ca-eed5b9481b7a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:43:43.288268 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.288236    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:43.292689 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.292657    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8494bcdf8c-w5qzf"]
Oct 29 22:43:43.292942 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.292926    5428 cpu_manager.go:414] "containerMap: removing container" podUID="5a381148-a1d6-404f-94ca-eed5b9481b7a" containerName="console"
Oct 29 22:43:43.292991 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.292945    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a381148-a1d6-404f-94ca-eed5b9481b7a" containerName="console"
Oct 29 22:43:43.292991 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.292954    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9e8347f7-98bc-4826-a26d-2ad74ec8b27e" containerName="console"
Oct 29 22:43:43.292991 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.292959    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8347f7-98bc-4826-a26d-2ad74ec8b27e" containerName="console"
Oct 29 22:43:43.293078 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.293011    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a381148-a1d6-404f-94ca-eed5b9481b7a" containerName="console"
Oct 29 22:43:43.293078 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.293021    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8347f7-98bc-4826-a26d-2ad74ec8b27e" containerName="console"
Oct 29 22:43:43.293224 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.293204    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtrgr\" (UniqueName: \"kubernetes.io/projected/5a381148-a1d6-404f-94ca-eed5b9481b7a-kube-api-access-gtrgr\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.293258 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.293229    5428 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-serving-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.293258 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.293238    5428 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-oauth-config\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.293258 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.293247    5428 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-oauth-serving-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.293258 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.293256    5428 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-config\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.293372 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.293265    5428 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-trusted-ca-bundle\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.293372 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.293274    5428 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a381148-a1d6-404f-94ca-eed5b9481b7a-console-serving-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.293372 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.293283    5428 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-config\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.293372 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.293291    5428 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-console-oauth-config\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.293372 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.293300    5428 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-oauth-serving-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.293372 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.293308    5428 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.293372 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.293316    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl249\" (UniqueName: \"kubernetes.io/projected/9e8347f7-98bc-4826-a26d-2ad74ec8b27e-kube-api-access-wl249\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.293372 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.293323    5428 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a381148-a1d6-404f-94ca-eed5b9481b7a-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:43:43.295831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.295795    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.329020 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.328957    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8494bcdf8c-w5qzf"]
Oct 29 22:43:43.393907 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.393872    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgnsg\" (UniqueName: \"kubernetes.io/projected/360d79bf-6e2e-4060-8318-f0d86c71c2e8-kube-api-access-xgnsg\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.393907 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.393909    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/360d79bf-6e2e-4060-8318-f0d86c71c2e8-console-serving-cert\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.394119 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.393926    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/360d79bf-6e2e-4060-8318-f0d86c71c2e8-console-config\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.394119 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.394074    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/360d79bf-6e2e-4060-8318-f0d86c71c2e8-console-oauth-config\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.394119 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.394115    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/360d79bf-6e2e-4060-8318-f0d86c71c2e8-trusted-ca-bundle\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.394252 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.394132    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/360d79bf-6e2e-4060-8318-f0d86c71c2e8-oauth-serving-cert\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.394252 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.394157    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/360d79bf-6e2e-4060-8318-f0d86c71c2e8-tmp\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.394252 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.394181    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/360d79bf-6e2e-4060-8318-f0d86c71c2e8-service-ca\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.428196 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.428166    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:43.495650 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.495617    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/360d79bf-6e2e-4060-8318-f0d86c71c2e8-console-oauth-config\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.495856 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.495661    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/360d79bf-6e2e-4060-8318-f0d86c71c2e8-trusted-ca-bundle\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.495856 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.495686    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/360d79bf-6e2e-4060-8318-f0d86c71c2e8-oauth-serving-cert\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.495856 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.495712    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/360d79bf-6e2e-4060-8318-f0d86c71c2e8-tmp\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.495856 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.495732    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/360d79bf-6e2e-4060-8318-f0d86c71c2e8-service-ca\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.495856 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.495753    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnsg\" (UniqueName: \"kubernetes.io/projected/360d79bf-6e2e-4060-8318-f0d86c71c2e8-kube-api-access-xgnsg\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.495856 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.495776    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/360d79bf-6e2e-4060-8318-f0d86c71c2e8-console-serving-cert\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.495856 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.495792    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/360d79bf-6e2e-4060-8318-f0d86c71c2e8-console-config\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.496212 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.496086    5428 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 22:43:43.496292 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.496267    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/360d79bf-6e2e-4060-8318-f0d86c71c2e8-tmp\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.496654 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.496631    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/360d79bf-6e2e-4060-8318-f0d86c71c2e8-console-config\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.496762 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.496671    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/360d79bf-6e2e-4060-8318-f0d86c71c2e8-service-ca\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.496762 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.496711    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/360d79bf-6e2e-4060-8318-f0d86c71c2e8-oauth-serving-cert\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.496907 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.496836    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/360d79bf-6e2e-4060-8318-f0d86c71c2e8-trusted-ca-bundle\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.498281 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.498243    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/360d79bf-6e2e-4060-8318-f0d86c71c2e8-console-oauth-config\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.498432 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.498413    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/360d79bf-6e2e-4060-8318-f0d86c71c2e8-console-serving-cert\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.516998 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.516977    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgnsg\" (UniqueName: \"kubernetes.io/projected/360d79bf-6e2e-4060-8318-f0d86c71c2e8-kube-api-access-xgnsg\") pod \"console-8494bcdf8c-w5qzf\" (UID: \"360d79bf-6e2e-4060-8318-f0d86c71c2e8\") " pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.553793 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.553761    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:43.605572 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.605498    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:43.903361 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.903327    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8494bcdf8c-w5qzf"]
Oct 29 22:43:43.908151 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:43:43.908120    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360d79bf_6e2e_4060_8318_f0d86c71c2e8.slice/crio-7e7c0ee4c690058f2bed19863c976458411a7b9d07ef152f9b4d8bd3e1be1df4 WatchSource:0}: Error finding container 7e7c0ee4c690058f2bed19863c976458411a7b9d07ef152f9b4d8bd3e1be1df4: Status 404 returned error can't find the container with id 7e7c0ee4c690058f2bed19863c976458411a7b9d07ef152f9b4d8bd3e1be1df4
Oct 29 22:43:43.909913 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.909887    5428 generic.go:356] "Generic (PLEG): container finished" podID="5a381148-a1d6-404f-94ca-eed5b9481b7a" containerID="08a02334529f0003ff8281388c584112623150bfc959c4a7ac12df4773c37929" exitCode=0
Oct 29 22:43:43.910016 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.909931    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bff8847db-vg5bs"
Oct 29 22:43:43.910016 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.909966    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bff8847db-vg5bs" event={"ID":"5a381148-a1d6-404f-94ca-eed5b9481b7a","Type":"ContainerDied","Data":"08a02334529f0003ff8281388c584112623150bfc959c4a7ac12df4773c37929"}
Oct 29 22:43:43.910016 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.910009    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bff8847db-vg5bs" event={"ID":"5a381148-a1d6-404f-94ca-eed5b9481b7a","Type":"ContainerDied","Data":"b8e96aa62ac1a645d730795eaee1426f93bf779657e28dfae2de6584d4698f47"}
Oct 29 22:43:43.910165 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.910036    5428 scope.go:122] "RemoveContainer" containerID="08a02334529f0003ff8281388c584112623150bfc959c4a7ac12df4773c37929"
Oct 29 22:43:43.911646 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.911622    5428 generic.go:356] "Generic (PLEG): container finished" podID="9e8347f7-98bc-4826-a26d-2ad74ec8b27e" containerID="acec23f1e3c79e2cfcf49ca0b451890c1b403af2ebe102a46cb2dcf9c835c289" exitCode=0
Oct 29 22:43:43.911749 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.911677    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56797698b7-xlsbd" event={"ID":"9e8347f7-98bc-4826-a26d-2ad74ec8b27e","Type":"ContainerDied","Data":"acec23f1e3c79e2cfcf49ca0b451890c1b403af2ebe102a46cb2dcf9c835c289"}
Oct 29 22:43:43.911749 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.911713    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56797698b7-xlsbd" event={"ID":"9e8347f7-98bc-4826-a26d-2ad74ec8b27e","Type":"ContainerDied","Data":"973fa4078033110d1cf17d12d7c74370d627c50584c08597c1dd0b031aeac820"}
Oct 29 22:43:43.911862 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.911776    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56797698b7-xlsbd"
Oct 29 22:43:43.920506 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.920491    5428 scope.go:122] "RemoveContainer" containerID="08a02334529f0003ff8281388c584112623150bfc959c4a7ac12df4773c37929"
Oct 29 22:43:43.920788 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:43:43.920768    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a02334529f0003ff8281388c584112623150bfc959c4a7ac12df4773c37929\": container with ID starting with 08a02334529f0003ff8281388c584112623150bfc959c4a7ac12df4773c37929 not found: ID does not exist" containerID="08a02334529f0003ff8281388c584112623150bfc959c4a7ac12df4773c37929"
Oct 29 22:43:43.920867 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.920796    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a02334529f0003ff8281388c584112623150bfc959c4a7ac12df4773c37929"} err="failed to get container status \"08a02334529f0003ff8281388c584112623150bfc959c4a7ac12df4773c37929\": rpc error: code = NotFound desc = could not find container \"08a02334529f0003ff8281388c584112623150bfc959c4a7ac12df4773c37929\": container with ID starting with 08a02334529f0003ff8281388c584112623150bfc959c4a7ac12df4773c37929 not found: ID does not exist"
Oct 29 22:43:43.920867 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.920847    5428 scope.go:122] "RemoveContainer" containerID="acec23f1e3c79e2cfcf49ca0b451890c1b403af2ebe102a46cb2dcf9c835c289"
Oct 29 22:43:43.933990 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.933971    5428 scope.go:122] "RemoveContainer" containerID="acec23f1e3c79e2cfcf49ca0b451890c1b403af2ebe102a46cb2dcf9c835c289"
Oct 29 22:43:43.934241 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:43:43.934221    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acec23f1e3c79e2cfcf49ca0b451890c1b403af2ebe102a46cb2dcf9c835c289\": container with ID starting with acec23f1e3c79e2cfcf49ca0b451890c1b403af2ebe102a46cb2dcf9c835c289 not found: ID does not exist" containerID="acec23f1e3c79e2cfcf49ca0b451890c1b403af2ebe102a46cb2dcf9c835c289"
Oct 29 22:43:43.934297 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.934248    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acec23f1e3c79e2cfcf49ca0b451890c1b403af2ebe102a46cb2dcf9c835c289"} err="failed to get container status \"acec23f1e3c79e2cfcf49ca0b451890c1b403af2ebe102a46cb2dcf9c835c289\": rpc error: code = NotFound desc = could not find container \"acec23f1e3c79e2cfcf49ca0b451890c1b403af2ebe102a46cb2dcf9c835c289\": container with ID starting with acec23f1e3c79e2cfcf49ca0b451890c1b403af2ebe102a46cb2dcf9c835c289 not found: ID does not exist"
Oct 29 22:43:43.968958 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.968923    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bff8847db-vg5bs"]
Oct 29 22:43:43.979031 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:43.979005    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bff8847db-vg5bs"]
Oct 29 22:43:44.011185 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:44.011144    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56797698b7-xlsbd"]
Oct 29 22:43:44.012575 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:44.012537    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56797698b7-xlsbd"]
Oct 29 22:43:44.049211 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:44.049178    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:44.158819 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:44.158733    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:44.508120 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:44.508042    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:44.615027 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:44.614993    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:44.773880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:44.773788    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a381148-a1d6-404f-94ca-eed5b9481b7a" path="/var/lib/kubelet/pods/5a381148-a1d6-404f-94ca-eed5b9481b7a/volumes"
Oct 29 22:43:44.774221 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:44.774208    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e8347f7-98bc-4826-a26d-2ad74ec8b27e" path="/var/lib/kubelet/pods/9e8347f7-98bc-4826-a26d-2ad74ec8b27e/volumes"
Oct 29 22:43:44.916307 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:44.916263    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8494bcdf8c-w5qzf" event={"ID":"360d79bf-6e2e-4060-8318-f0d86c71c2e8","Type":"ContainerStarted","Data":"eaa4a7ae982f79f5dfde2b0e28106e2969d0bfb4e928b33f05400539f0a2deca"}
Oct 29 22:43:44.916740 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:44.916324    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8494bcdf8c-w5qzf" event={"ID":"360d79bf-6e2e-4060-8318-f0d86c71c2e8","Type":"ContainerStarted","Data":"7e7c0ee4c690058f2bed19863c976458411a7b9d07ef152f9b4d8bd3e1be1df4"}
Oct 29 22:43:44.965614 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:44.965552    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8494bcdf8c-w5qzf" podStartSLOduration=1.965538359 podStartE2EDuration="1.965538359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:44.955302397 +0000 UTC m=+28.298251142" watchObservedRunningTime="2026-10-29 22:43:44.965538359 +0000 UTC m=+28.308487100"
Oct 29 22:43:46.291682 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:46.291650    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:46.407948 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:46.407916    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:47.372655 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:43:47.372618    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:43:52.853770 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:52.853739    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lqjl6"
Oct 29 22:43:53.605767 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:53.605732    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:53.605937 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:53.605821    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:53.610536 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:53.610510    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:53.952114 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:53.952088    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8494bcdf8c-w5qzf"
Oct 29 22:43:54.020585 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:54.020553    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-774f56585c-2x9nc"]
Oct 29 22:43:59.556393 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:59.556357    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:59.678547 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:43:59.678516    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:17.364695 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:44:17.364654    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:44:19.045056 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.045017    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-774f56585c-2x9nc" podUID="d31c1857-282f-4a04-89a9-883b1c2a7c19" containerName="console" containerID="cri-o://7bf853c9ce736e09b91d80c35913f123d769773021c0b64317798aab342c812f" gracePeriod=15
Oct 29 22:44:19.282299 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.282269    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:44:19.357539 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.357473    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-serving-cert\") pod \"d31c1857-282f-4a04-89a9-883b1c2a7c19\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") "
Oct 29 22:44:19.357539 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.357506    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-config\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-config\") pod \"d31c1857-282f-4a04-89a9-883b1c2a7c19\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") "
Oct 29 22:44:19.357539 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.357523    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-trusted-ca-bundle\") pod \"d31c1857-282f-4a04-89a9-883b1c2a7c19\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") "
Oct 29 22:44:19.357539 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.357540    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-service-ca\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-service-ca\") pod \"d31c1857-282f-4a04-89a9-883b1c2a7c19\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") "
Oct 29 22:44:19.357868 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.357565    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-oauth-serving-cert\") pod \"d31c1857-282f-4a04-89a9-883b1c2a7c19\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") "
Oct 29 22:44:19.357868 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.357587    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31c1857-282f-4a04-89a9-883b1c2a7c19-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d31c1857-282f-4a04-89a9-883b1c2a7c19-tmp\") pod \"d31c1857-282f-4a04-89a9-883b1c2a7c19\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") "
Oct 29 22:44:19.357868 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.357643    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-oauth-config\") pod \"d31c1857-282f-4a04-89a9-883b1c2a7c19\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") "
Oct 29 22:44:19.357868 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.357671    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31c1857-282f-4a04-89a9-883b1c2a7c19-kube-api-access-grckl\" (UniqueName: \"kubernetes.io/projected/d31c1857-282f-4a04-89a9-883b1c2a7c19-kube-api-access-grckl\") pod \"d31c1857-282f-4a04-89a9-883b1c2a7c19\" (UID: \"d31c1857-282f-4a04-89a9-883b1c2a7c19\") "
Oct 29 22:44:19.358110 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.357928    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31c1857-282f-4a04-89a9-883b1c2a7c19-tmp" pod "d31c1857-282f-4a04-89a9-883b1c2a7c19" (UID: "d31c1857-282f-4a04-89a9-883b1c2a7c19"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:19.358110 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.357958    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-config" pod "d31c1857-282f-4a04-89a9-883b1c2a7c19" (UID: "d31c1857-282f-4a04-89a9-883b1c2a7c19"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:44:19.358110 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.357981    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-trusted-ca-bundle" pod "d31c1857-282f-4a04-89a9-883b1c2a7c19" (UID: "d31c1857-282f-4a04-89a9-883b1c2a7c19"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:44:19.358110 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.358017    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-oauth-serving-cert" pod "d31c1857-282f-4a04-89a9-883b1c2a7c19" (UID: "d31c1857-282f-4a04-89a9-883b1c2a7c19"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:44:19.358110 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.358035    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-service-ca" pod "d31c1857-282f-4a04-89a9-883b1c2a7c19" (UID: "d31c1857-282f-4a04-89a9-883b1c2a7c19"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:44:19.359693 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.359668    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-serving-cert" pod "d31c1857-282f-4a04-89a9-883b1c2a7c19" (UID: "d31c1857-282f-4a04-89a9-883b1c2a7c19"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:44:19.359793 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.359716    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31c1857-282f-4a04-89a9-883b1c2a7c19-kube-api-access-grckl" pod "d31c1857-282f-4a04-89a9-883b1c2a7c19" (UID: "d31c1857-282f-4a04-89a9-883b1c2a7c19"). InnerVolumeSpecName "kube-api-access-grckl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:44:19.359873 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.359848    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-oauth-config" pod "d31c1857-282f-4a04-89a9-883b1c2a7c19" (UID: "d31c1857-282f-4a04-89a9-883b1c2a7c19"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:44:19.458224 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.458184    5428 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-serving-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:44:19.458224 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.458218    5428 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-config\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:44:19.458224 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.458228    5428 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-trusted-ca-bundle\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:44:19.458436 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.458236    5428 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:44:19.458436 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.458245    5428 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d31c1857-282f-4a04-89a9-883b1c2a7c19-oauth-serving-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:44:19.458436 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.458254    5428 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d31c1857-282f-4a04-89a9-883b1c2a7c19-tmp\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:44:19.458436 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.458266    5428 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d31c1857-282f-4a04-89a9-883b1c2a7c19-console-oauth-config\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:44:19.458436 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:19.458280    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grckl\" (UniqueName: \"kubernetes.io/projected/d31c1857-282f-4a04-89a9-883b1c2a7c19-kube-api-access-grckl\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:44:20.034330 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:20.034286    5428 generic.go:356] "Generic (PLEG): container finished" podID="d31c1857-282f-4a04-89a9-883b1c2a7c19" containerID="7bf853c9ce736e09b91d80c35913f123d769773021c0b64317798aab342c812f" exitCode=0
Oct 29 22:44:20.034491 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:20.034339    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774f56585c-2x9nc"
Oct 29 22:44:20.034491 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:20.034339    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774f56585c-2x9nc" event={"ID":"d31c1857-282f-4a04-89a9-883b1c2a7c19","Type":"ContainerDied","Data":"7bf853c9ce736e09b91d80c35913f123d769773021c0b64317798aab342c812f"}
Oct 29 22:44:20.034491 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:20.034477    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774f56585c-2x9nc" event={"ID":"d31c1857-282f-4a04-89a9-883b1c2a7c19","Type":"ContainerDied","Data":"f5fa9216f177e87c477e1263a558f14e20de1af6f27bba84a2fe048320d3ff15"}
Oct 29 22:44:20.034608 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:20.034505    5428 scope.go:122] "RemoveContainer" containerID="7bf853c9ce736e09b91d80c35913f123d769773021c0b64317798aab342c812f"
Oct 29 22:44:20.043178 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:20.043160    5428 scope.go:122] "RemoveContainer" containerID="7bf853c9ce736e09b91d80c35913f123d769773021c0b64317798aab342c812f"
Oct 29 22:44:20.043439 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:44:20.043421    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bf853c9ce736e09b91d80c35913f123d769773021c0b64317798aab342c812f\": container with ID starting with 7bf853c9ce736e09b91d80c35913f123d769773021c0b64317798aab342c812f not found: ID does not exist" containerID="7bf853c9ce736e09b91d80c35913f123d769773021c0b64317798aab342c812f"
Oct 29 22:44:20.043505 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:20.043453    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bf853c9ce736e09b91d80c35913f123d769773021c0b64317798aab342c812f"} err="failed to get container status \"7bf853c9ce736e09b91d80c35913f123d769773021c0b64317798aab342c812f\": rpc error: code = NotFound desc = could not find container \"7bf853c9ce736e09b91d80c35913f123d769773021c0b64317798aab342c812f\": container with ID starting with 7bf853c9ce736e09b91d80c35913f123d769773021c0b64317798aab342c812f not found: ID does not exist"
Oct 29 22:44:20.082632 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:20.082598    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-774f56585c-2x9nc"]
Oct 29 22:44:20.093025 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:20.092998    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-774f56585c-2x9nc"]
Oct 29 22:44:20.772737 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:20.772703    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d31c1857-282f-4a04-89a9-883b1c2a7c19" path="/var/lib/kubelet/pods/d31c1857-282f-4a04-89a9-883b1c2a7c19/volumes"
Oct 29 22:44:22.727285 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:22.727238    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:27.476059 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:44:27.476030    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 22:44:47.365167 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:44:47.365132    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:45:10.713554 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:10.713521    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:12.066573 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:12.066511    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:17.369071 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:45:17.369022    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:45:22.831539 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:22.831501    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:39.712085 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:39.712047    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:39.892392 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:39.892365    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:41.279787 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:41.279752    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:41.398417 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:41.398382    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:42.671951 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:42.671917    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:43.722574 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:43.722530    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:43.843532 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:43.843501    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:44.656197 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.656159    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc"]
Oct 29 22:45:44.656495 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.656474    5428 cpu_manager.go:414] "containerMap: removing container" podUID="d31c1857-282f-4a04-89a9-883b1c2a7c19" containerName="console"
Oct 29 22:45:44.656620 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.656599    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d31c1857-282f-4a04-89a9-883b1c2a7c19" containerName="console"
Oct 29 22:45:44.656774 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.656709    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d31c1857-282f-4a04-89a9-883b1c2a7c19" containerName="console"
Oct 29 22:45:44.658859 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.658836    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc"
Oct 29 22:45:44.682093 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.682069    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-n2nk9\""
Oct 29 22:45:44.682321 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.682125    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 29 22:45:44.682321 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.682140    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 29 22:45:44.683070 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.683023    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc"]
Oct 29 22:45:44.756680 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.756648    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc\" (UID: \"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc"
Oct 29 22:45:44.757071 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.756692    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd86q\" (UniqueName: \"kubernetes.io/projected/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-kube-api-access-hd86q\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc\" (UID: \"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc"
Oct 29 22:45:44.757071 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.756820    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc\" (UID: \"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc"
Oct 29 22:45:44.857375 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.857336    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc\" (UID: \"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc"
Oct 29 22:45:44.857375 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.857377    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd86q\" (UniqueName: \"kubernetes.io/projected/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-kube-api-access-hd86q\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc\" (UID: \"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc"
Oct 29 22:45:44.857635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.857508    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc\" (UID: \"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc"
Oct 29 22:45:44.857835 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.857775    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc\" (UID: \"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc"
Oct 29 22:45:44.857835 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.857821    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc\" (UID: \"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc"
Oct 29 22:45:44.879686 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.879657    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd86q\" (UniqueName: \"kubernetes.io/projected/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-kube-api-access-hd86q\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc\" (UID: \"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc"
Oct 29 22:45:44.971711 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.971646    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc"
Oct 29 22:45:44.980156 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.980132    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d"]
Oct 29 22:45:44.982612 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.982593    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d"
Oct 29 22:45:44.995454 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:44.995429    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-d2mp6\""
Oct 29 22:45:45.005422 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.005332    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d"]
Oct 29 22:45:45.059298 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.059273    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86n8x\" (UniqueName: \"kubernetes.io/projected/ae7e30b7-96de-442d-bd5b-149161c4df5b-kube-api-access-86n8x\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d\" (UID: \"ae7e30b7-96de-442d-bd5b-149161c4df5b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d"
Oct 29 22:45:45.059399 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.059318    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae7e30b7-96de-442d-bd5b-149161c4df5b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d\" (UID: \"ae7e30b7-96de-442d-bd5b-149161c4df5b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d"
Oct 29 22:45:45.059503 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.059481    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae7e30b7-96de-442d-bd5b-149161c4df5b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d\" (UID: \"ae7e30b7-96de-442d-bd5b-149161c4df5b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d"
Oct 29 22:45:45.076366 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.076345    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8"]
Oct 29 22:45:45.078627 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.078609    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8"
Oct 29 22:45:45.091393 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.091372    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-svh9x\""
Oct 29 22:45:45.100916 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.100892    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8"]
Oct 29 22:45:45.160262 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.160232    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44c71a21-a3e3-4344-b32b-b39b09a7a440-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8\" (UID: \"44c71a21-a3e3-4344-b32b-b39b09a7a440\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8"
Oct 29 22:45:45.160262 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.160264    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsbwk\" (UniqueName: \"kubernetes.io/projected/44c71a21-a3e3-4344-b32b-b39b09a7a440-kube-api-access-nsbwk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8\" (UID: \"44c71a21-a3e3-4344-b32b-b39b09a7a440\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8"
Oct 29 22:45:45.160440 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.160298    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86n8x\" (UniqueName: \"kubernetes.io/projected/ae7e30b7-96de-442d-bd5b-149161c4df5b-kube-api-access-86n8x\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d\" (UID: \"ae7e30b7-96de-442d-bd5b-149161c4df5b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d"
Oct 29 22:45:45.160561 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.160534    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae7e30b7-96de-442d-bd5b-149161c4df5b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d\" (UID: \"ae7e30b7-96de-442d-bd5b-149161c4df5b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d"
Oct 29 22:45:45.160599 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.160571    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44c71a21-a3e3-4344-b32b-b39b09a7a440-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8\" (UID: \"44c71a21-a3e3-4344-b32b-b39b09a7a440\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8"
Oct 29 22:45:45.160641 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.160599    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae7e30b7-96de-442d-bd5b-149161c4df5b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d\" (UID: \"ae7e30b7-96de-442d-bd5b-149161c4df5b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d"
Oct 29 22:45:45.160945 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.160926    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae7e30b7-96de-442d-bd5b-149161c4df5b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d\" (UID: \"ae7e30b7-96de-442d-bd5b-149161c4df5b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d"
Oct 29 22:45:45.160992 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.160956    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae7e30b7-96de-442d-bd5b-149161c4df5b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d\" (UID: \"ae7e30b7-96de-442d-bd5b-149161c4df5b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d"
Oct 29 22:45:45.176299 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.176276    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx"]
Oct 29 22:45:45.178502 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.178486    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx"
Oct 29 22:45:45.186207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.186183    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86n8x\" (UniqueName: \"kubernetes.io/projected/ae7e30b7-96de-442d-bd5b-149161c4df5b-kube-api-access-86n8x\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d\" (UID: \"ae7e30b7-96de-442d-bd5b-149161c4df5b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d"
Oct 29 22:45:45.191433 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.191409    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-r5cxk\""
Oct 29 22:45:45.200541 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.200519    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx"]
Oct 29 22:45:45.261594 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.261528    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kvx7\" (UniqueName: \"kubernetes.io/projected/edb57588-cd4b-4881-8976-d45d43474ba6-kube-api-access-2kvx7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx\" (UID: \"edb57588-cd4b-4881-8976-d45d43474ba6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx"
Oct 29 22:45:45.261594 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.261578    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44c71a21-a3e3-4344-b32b-b39b09a7a440-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8\" (UID: \"44c71a21-a3e3-4344-b32b-b39b09a7a440\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8"
Oct 29 22:45:45.261745 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.261650    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsbwk\" (UniqueName: \"kubernetes.io/projected/44c71a21-a3e3-4344-b32b-b39b09a7a440-kube-api-access-nsbwk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8\" (UID: \"44c71a21-a3e3-4344-b32b-b39b09a7a440\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8"
Oct 29 22:45:45.261745 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.261691    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edb57588-cd4b-4881-8976-d45d43474ba6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx\" (UID: \"edb57588-cd4b-4881-8976-d45d43474ba6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx"
Oct 29 22:45:45.261835 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.261745    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44c71a21-a3e3-4344-b32b-b39b09a7a440-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8\" (UID: \"44c71a21-a3e3-4344-b32b-b39b09a7a440\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8"
Oct 29 22:45:45.261835 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.261819    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edb57588-cd4b-4881-8976-d45d43474ba6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx\" (UID: \"edb57588-cd4b-4881-8976-d45d43474ba6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx"
Oct 29 22:45:45.261992 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.261976    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44c71a21-a3e3-4344-b32b-b39b09a7a440-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8\" (UID: \"44c71a21-a3e3-4344-b32b-b39b09a7a440\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8"
Oct 29 22:45:45.262077 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.262060    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44c71a21-a3e3-4344-b32b-b39b09a7a440-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8\" (UID: \"44c71a21-a3e3-4344-b32b-b39b09a7a440\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8"
Oct 29 22:45:45.269101 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.269071    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc"]
Oct 29 22:45:45.273992 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:45:45.273971    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea52eb5_27d8_4ae6_adac_80fc3efb3f40.slice/crio-1ad4fcc665dc89b116eacab5a7061383402ff5fc228b8456a9ba166a3696341a WatchSource:0}: Error finding container 1ad4fcc665dc89b116eacab5a7061383402ff5fc228b8456a9ba166a3696341a: Status 404 returned error can't find the container with id 1ad4fcc665dc89b116eacab5a7061383402ff5fc228b8456a9ba166a3696341a
Oct 29 22:45:45.276436 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.276416    5428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:45:45.283616 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.283594    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsbwk\" (UniqueName: \"kubernetes.io/projected/44c71a21-a3e3-4344-b32b-b39b09a7a440-kube-api-access-nsbwk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8\" (UID: \"44c71a21-a3e3-4344-b32b-b39b09a7a440\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8"
Oct 29 22:45:45.286772 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.286747    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59"]
Oct 29 22:45:45.289586 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.289569    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59"
Oct 29 22:45:45.300904 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.300886    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d"
Oct 29 22:45:45.302538 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.302494    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-rb77k\""
Oct 29 22:45:45.307409 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.307390    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc" event={"ID":"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40","Type":"ContainerStarted","Data":"1ad4fcc665dc89b116eacab5a7061383402ff5fc228b8456a9ba166a3696341a"}
Oct 29 22:45:45.312375 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.312350    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59"]
Oct 29 22:45:45.362211 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.362186    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb43c41a-587f-47d6-affe-3231ce614d8d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59\" (UID: \"fb43c41a-587f-47d6-affe-3231ce614d8d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59"
Oct 29 22:45:45.362323 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.362223    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edb57588-cd4b-4881-8976-d45d43474ba6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx\" (UID: \"edb57588-cd4b-4881-8976-d45d43474ba6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx"
Oct 29 22:45:45.362366 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.362321    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kvx7\" (UniqueName: \"kubernetes.io/projected/edb57588-cd4b-4881-8976-d45d43474ba6-kube-api-access-2kvx7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx\" (UID: \"edb57588-cd4b-4881-8976-d45d43474ba6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx"
Oct 29 22:45:45.362400 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.362364    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb43c41a-587f-47d6-affe-3231ce614d8d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59\" (UID: \"fb43c41a-587f-47d6-affe-3231ce614d8d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59"
Oct 29 22:45:45.362432 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.362416    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edb57588-cd4b-4881-8976-d45d43474ba6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx\" (UID: \"edb57588-cd4b-4881-8976-d45d43474ba6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx"
Oct 29 22:45:45.362466 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.362441    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx75t\" (UniqueName: \"kubernetes.io/projected/fb43c41a-587f-47d6-affe-3231ce614d8d-kube-api-access-cx75t\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59\" (UID: \"fb43c41a-587f-47d6-affe-3231ce614d8d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59"
Oct 29 22:45:45.362687 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.362670    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edb57588-cd4b-4881-8976-d45d43474ba6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx\" (UID: \"edb57588-cd4b-4881-8976-d45d43474ba6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx"
Oct 29 22:45:45.362746 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.362705    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edb57588-cd4b-4881-8976-d45d43474ba6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx\" (UID: \"edb57588-cd4b-4881-8976-d45d43474ba6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx"
Oct 29 22:45:45.384073 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.384049    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kvx7\" (UniqueName: \"kubernetes.io/projected/edb57588-cd4b-4881-8976-d45d43474ba6-kube-api-access-2kvx7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx\" (UID: \"edb57588-cd4b-4881-8976-d45d43474ba6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx"
Oct 29 22:45:45.390864 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.390847    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8"
Oct 29 22:45:45.463290 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.463254    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb43c41a-587f-47d6-affe-3231ce614d8d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59\" (UID: \"fb43c41a-587f-47d6-affe-3231ce614d8d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59"
Oct 29 22:45:45.463425 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.463341    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb43c41a-587f-47d6-affe-3231ce614d8d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59\" (UID: \"fb43c41a-587f-47d6-affe-3231ce614d8d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59"
Oct 29 22:45:45.463425 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.463393    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx75t\" (UniqueName: \"kubernetes.io/projected/fb43c41a-587f-47d6-affe-3231ce614d8d-kube-api-access-cx75t\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59\" (UID: \"fb43c41a-587f-47d6-affe-3231ce614d8d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59"
Oct 29 22:45:45.463716 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.463690    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb43c41a-587f-47d6-affe-3231ce614d8d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59\" (UID: \"fb43c41a-587f-47d6-affe-3231ce614d8d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59"
Oct 29 22:45:45.463716 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.463708    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb43c41a-587f-47d6-affe-3231ce614d8d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59\" (UID: \"fb43c41a-587f-47d6-affe-3231ce614d8d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59"
Oct 29 22:45:45.485401 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.485376    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx75t\" (UniqueName: \"kubernetes.io/projected/fb43c41a-587f-47d6-affe-3231ce614d8d-kube-api-access-cx75t\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59\" (UID: \"fb43c41a-587f-47d6-affe-3231ce614d8d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59"
Oct 29 22:45:45.490205 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.490187    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx"
Oct 29 22:45:45.596152 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.596120    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d"]
Oct 29 22:45:45.601313 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.601281    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59"
Oct 29 22:45:45.601462 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:45:45.601427    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7e30b7_96de_442d_bd5b_149161c4df5b.slice/crio-01982a0fbbe250dc328109d35039cbd89302454b631638c1af79dc6d0247f09c WatchSource:0}: Error finding container 01982a0fbbe250dc328109d35039cbd89302454b631638c1af79dc6d0247f09c: Status 404 returned error can't find the container with id 01982a0fbbe250dc328109d35039cbd89302454b631638c1af79dc6d0247f09c
Oct 29 22:45:45.684966 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.684887    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8"]
Oct 29 22:45:45.690577 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:45:45.690551    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c71a21_a3e3_4344_b32b_b39b09a7a440.slice/crio-d751865b6d753e1291a2b784ffd2a23f2b097ce5dea4f68dbe2a9b70abbe1c59 WatchSource:0}: Error finding container d751865b6d753e1291a2b784ffd2a23f2b097ce5dea4f68dbe2a9b70abbe1c59: Status 404 returned error can't find the container with id d751865b6d753e1291a2b784ffd2a23f2b097ce5dea4f68dbe2a9b70abbe1c59
Oct 29 22:45:45.784317 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.784250    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx"]
Oct 29 22:45:45.789254 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:45:45.789227    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb57588_cd4b_4881_8976_d45d43474ba6.slice/crio-b4f8b4973489fa881813e025579c903c2647a6f4990c19d00e895f3b3e9f8960 WatchSource:0}: Error finding container b4f8b4973489fa881813e025579c903c2647a6f4990c19d00e895f3b3e9f8960: Status 404 returned error can't find the container with id b4f8b4973489fa881813e025579c903c2647a6f4990c19d00e895f3b3e9f8960
Oct 29 22:45:45.901747 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:45.901711    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59"]
Oct 29 22:45:45.907166 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:45:45.907139    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb43c41a_587f_47d6_affe_3231ce614d8d.slice/crio-d640da4dd36938d09a3112653094321fbc2f23907fde43a9fd86c34d3d71e150 WatchSource:0}: Error finding container d640da4dd36938d09a3112653094321fbc2f23907fde43a9fd86c34d3d71e150: Status 404 returned error can't find the container with id d640da4dd36938d09a3112653094321fbc2f23907fde43a9fd86c34d3d71e150
Oct 29 22:45:46.295344 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:46.295295    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:46.314587 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:46.314461    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8" event={"ID":"44c71a21-a3e3-4344-b32b-b39b09a7a440","Type":"ContainerStarted","Data":"d751865b6d753e1291a2b784ffd2a23f2b097ce5dea4f68dbe2a9b70abbe1c59"}
Oct 29 22:45:46.317156 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:46.317098    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx" event={"ID":"edb57588-cd4b-4881-8976-d45d43474ba6","Type":"ContainerStarted","Data":"b4f8b4973489fa881813e025579c903c2647a6f4990c19d00e895f3b3e9f8960"}
Oct 29 22:45:46.318712 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:46.318620    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59" event={"ID":"fb43c41a-587f-47d6-affe-3231ce614d8d","Type":"ContainerStarted","Data":"d640da4dd36938d09a3112653094321fbc2f23907fde43a9fd86c34d3d71e150"}
Oct 29 22:45:46.320492 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:46.320473    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d" event={"ID":"ae7e30b7-96de-442d-bd5b-149161c4df5b","Type":"ContainerStarted","Data":"01982a0fbbe250dc328109d35039cbd89302454b631638c1af79dc6d0247f09c"}
Oct 29 22:45:46.417609 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:46.417509    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:47.272347 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.271055    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g"]
Oct 29 22:45:47.274625 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.273938    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g"
Oct 29 22:45:47.298027 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.297999    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-g6z8r\""
Oct 29 22:45:47.304045 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.304019    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g"]
Oct 29 22:45:47.381161 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.381133    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/958235d6-aaec-45bf-a478-4ac4e20a272e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g\" (UID: \"958235d6-aaec-45bf-a478-4ac4e20a272e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g"
Oct 29 22:45:47.381346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.381206    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r87j\" (UniqueName: \"kubernetes.io/projected/958235d6-aaec-45bf-a478-4ac4e20a272e-kube-api-access-9r87j\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g\" (UID: \"958235d6-aaec-45bf-a478-4ac4e20a272e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g"
Oct 29 22:45:47.381346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.381244    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/958235d6-aaec-45bf-a478-4ac4e20a272e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g\" (UID: \"958235d6-aaec-45bf-a478-4ac4e20a272e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g"
Oct 29 22:45:47.384752 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:45:47.384679    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:45:47.483261 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.482224    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/958235d6-aaec-45bf-a478-4ac4e20a272e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g\" (UID: \"958235d6-aaec-45bf-a478-4ac4e20a272e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g"
Oct 29 22:45:47.483261 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.482314    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r87j\" (UniqueName: \"kubernetes.io/projected/958235d6-aaec-45bf-a478-4ac4e20a272e-kube-api-access-9r87j\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g\" (UID: \"958235d6-aaec-45bf-a478-4ac4e20a272e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g"
Oct 29 22:45:47.483261 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.482362    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/958235d6-aaec-45bf-a478-4ac4e20a272e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g\" (UID: \"958235d6-aaec-45bf-a478-4ac4e20a272e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g"
Oct 29 22:45:47.483261 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.482787    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/958235d6-aaec-45bf-a478-4ac4e20a272e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g\" (UID: \"958235d6-aaec-45bf-a478-4ac4e20a272e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g"
Oct 29 22:45:47.483261 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.482992    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/958235d6-aaec-45bf-a478-4ac4e20a272e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g\" (UID: \"958235d6-aaec-45bf-a478-4ac4e20a272e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g"
Oct 29 22:45:47.517776 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.517743    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r87j\" (UniqueName: \"kubernetes.io/projected/958235d6-aaec-45bf-a478-4ac4e20a272e-kube-api-access-9r87j\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g\" (UID: \"958235d6-aaec-45bf-a478-4ac4e20a272e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g"
Oct 29 22:45:47.595059 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.594986    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g"
Oct 29 22:45:47.667481 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.666014    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"]
Oct 29 22:45:47.680222 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.679100    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.696827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.696762    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"]
Oct 29 22:45:47.697157 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.697048    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 29 22:45:47.697157 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.697088    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 29 22:45:47.697157 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.697129    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 29 22:45:47.697337 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.697185    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h76hm\""
Oct 29 22:45:47.743336 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.743206    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:47.787619 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.787119    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.787619 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.787155    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.787619 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.787190    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv89j\" (UniqueName: \"kubernetes.io/projected/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-kube-api-access-rv89j\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.787619 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.787221    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.787619 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.787246    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.787619 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.787274    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.787619 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.787309    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.787619 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.787346    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.787619 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.787383    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.889177 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.888377    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.889177 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.888435    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.889177 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.888484    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.889177 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.888525    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.889177 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.888553    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.889177 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.888592    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.889177 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.888627    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.889177 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.888651    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.889783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.889717    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.890000 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.889971    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.890126 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.890110    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.890388 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.890365    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.890473 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.890446    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv89j\" (UniqueName: \"kubernetes.io/projected/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-kube-api-access-rv89j\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.890989 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.890963    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.893027 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.893003    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.911932 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.911880    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.917436 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.917412    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.923322 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.923298    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv89j\" (UniqueName: \"kubernetes.io/projected/fbea7545-5d53-4ba1-bb53-584acf2a2bc7-kube-api-access-rv89j\") pod \"maas-default-gateway-openshift-default-55fbd65575-lbp8m\" (UID: \"fbea7545-5d53-4ba1-bb53-584acf2a2bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:47.930262 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.930227    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:47.941775 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:47.941683    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g"]
Oct 29 22:45:48.001154 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:48.000941    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:48.054713 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:48.054671    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:48.335582 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:48.335502    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g" event={"ID":"958235d6-aaec-45bf-a478-4ac4e20a272e","Type":"ContainerStarted","Data":"42d39dddaf64bdd45c802cdf06d1e099b80ba37938891d76185debfab0d83a82"}
Oct 29 22:45:48.336318 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:48.336293    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"]
Oct 29 22:45:48.343164 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:45:48.343134    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbea7545_5d53_4ba1_bb53_584acf2a2bc7.slice/crio-6e3b284dd317cf1152c59720bd84f9b9cd2f179b39c4be9f606e4536b87fe9ba WatchSource:0}: Error finding container 6e3b284dd317cf1152c59720bd84f9b9cd2f179b39c4be9f606e4536b87fe9ba: Status 404 returned error can't find the container with id 6e3b284dd317cf1152c59720bd84f9b9cd2f179b39c4be9f606e4536b87fe9ba
Oct 29 22:45:48.702102 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:48.702063    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb"]
Oct 29 22:45:48.706007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:48.705274    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb"
Oct 29 22:45:48.719299 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:48.719249    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-rvjfw\""
Oct 29 22:45:48.729819 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:48.729781    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb"]
Oct 29 22:45:48.800951 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:48.800686    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/133574bb-7b45-4950-b58b-758c5fff1df1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb\" (UID: \"133574bb-7b45-4950-b58b-758c5fff1df1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb"
Oct 29 22:45:48.800951 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:48.800736    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/133574bb-7b45-4950-b58b-758c5fff1df1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb\" (UID: \"133574bb-7b45-4950-b58b-758c5fff1df1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb"
Oct 29 22:45:48.800951 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:48.800775    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h42dv\" (UniqueName: \"kubernetes.io/projected/133574bb-7b45-4950-b58b-758c5fff1df1-kube-api-access-h42dv\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb\" (UID: \"133574bb-7b45-4950-b58b-758c5fff1df1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb"
Oct 29 22:45:48.903516 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:48.902327    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h42dv\" (UniqueName: \"kubernetes.io/projected/133574bb-7b45-4950-b58b-758c5fff1df1-kube-api-access-h42dv\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb\" (UID: \"133574bb-7b45-4950-b58b-758c5fff1df1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb"
Oct 29 22:45:48.903516 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:48.902451    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/133574bb-7b45-4950-b58b-758c5fff1df1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb\" (UID: \"133574bb-7b45-4950-b58b-758c5fff1df1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb"
Oct 29 22:45:48.903516 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:48.902478    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/133574bb-7b45-4950-b58b-758c5fff1df1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb\" (UID: \"133574bb-7b45-4950-b58b-758c5fff1df1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb"
Oct 29 22:45:48.903516 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:48.902909    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/133574bb-7b45-4950-b58b-758c5fff1df1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb\" (UID: \"133574bb-7b45-4950-b58b-758c5fff1df1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb"
Oct 29 22:45:48.903516 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:48.903457    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/133574bb-7b45-4950-b58b-758c5fff1df1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb\" (UID: \"133574bb-7b45-4950-b58b-758c5fff1df1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb"
Oct 29 22:45:48.930425 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:48.930353    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h42dv\" (UniqueName: \"kubernetes.io/projected/133574bb-7b45-4950-b58b-758c5fff1df1-kube-api-access-h42dv\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb\" (UID: \"133574bb-7b45-4950-b58b-758c5fff1df1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb"
Oct 29 22:45:49.023566 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:49.023257    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb"
Oct 29 22:45:49.355121 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:49.354325    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m" event={"ID":"fbea7545-5d53-4ba1-bb53-584acf2a2bc7","Type":"ContainerStarted","Data":"6e3b284dd317cf1152c59720bd84f9b9cd2f179b39c4be9f606e4536b87fe9ba"}
Oct 29 22:45:49.400412 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:49.400377    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb"]
Oct 29 22:45:50.361836 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:50.361777    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb" event={"ID":"133574bb-7b45-4950-b58b-758c5fff1df1","Type":"ContainerStarted","Data":"f7482bdc0e7440cd200ae42393ecc043b1c7b720e1c15bf7cbc2aa41731e65b5"}
Oct 29 22:45:51.342429 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:51.342332    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:53.631151 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:53.631047    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:55.215190 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:55.214959    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:45:55.215190 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:55.215039    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:45:55.215190 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:55.215092    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:45:55.407027 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:55.406790    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m" event={"ID":"fbea7545-5d53-4ba1-bb53-584acf2a2bc7","Type":"ContainerStarted","Data":"c11fd48fad10673ac1337687063ab5412cda3e6c09e78e7f5959ed829bfc633c"}
Oct 29 22:45:55.483082 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:55.482964    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m" podStartSLOduration=1.61591409 podStartE2EDuration="8.482946405s" totalImagesPullingTime="6.867032315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:55.472364739 +0000 UTC m=+158.815313496" watchObservedRunningTime="2026-10-29 22:45:55.482946405 +0000 UTC m=+158.825895147"
Oct 29 22:45:56.002253 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:56.002211    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:56.008299 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:56.008263    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:56.413009 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:56.412948    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:56.414408 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:56.414200    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:45:59.436294 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:59.436216    5428 generic.go:356] "Generic (PLEG): container finished" podID="5ea52eb5-27d8-4ae6-adac-80fc3efb3f40" containerID="34e4917977e2d0625665ce6e36bf893c90e0c5d94c1b78306c091e718e208494" exitCode=0
Oct 29 22:45:59.436294 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:45:59.436288    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc" event={"ID":"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40","Type":"ContainerDied","Data":"34e4917977e2d0625665ce6e36bf893c90e0c5d94c1b78306c091e718e208494"}
Oct 29 22:46:00.443206 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:00.443168    5428 generic.go:356] "Generic (PLEG): container finished" podID="5ea52eb5-27d8-4ae6-adac-80fc3efb3f40" containerID="1ddea8225367355660b2073b2e48d7006b52b314a56f18b028d69ded51859b45" exitCode=0
Oct 29 22:46:00.443691 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:00.443456    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc" event={"ID":"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40","Type":"ContainerDied","Data":"1ddea8225367355660b2073b2e48d7006b52b314a56f18b028d69ded51859b45"}
Oct 29 22:46:00.446237 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:00.446217    5428 generic.go:356] "Generic (PLEG): container finished" podID="ae7e30b7-96de-442d-bd5b-149161c4df5b" containerID="afb50bc5f8a8498ffebb811f93b0c85df4d8f6b3f294c18eb88ff86f5c744f5e" exitCode=0
Oct 29 22:46:00.446346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:00.446326    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d" event={"ID":"ae7e30b7-96de-442d-bd5b-149161c4df5b","Type":"ContainerDied","Data":"afb50bc5f8a8498ffebb811f93b0c85df4d8f6b3f294c18eb88ff86f5c744f5e"}
Oct 29 22:46:00.449195 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:00.449171    5428 generic.go:356] "Generic (PLEG): container finished" podID="edb57588-cd4b-4881-8976-d45d43474ba6" containerID="f6f1afdb996d96eaa5ebaab0cca083c626c35eede28d77574a289826f69a8d9b" exitCode=0
Oct 29 22:46:00.449422 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:00.449403    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx" event={"ID":"edb57588-cd4b-4881-8976-d45d43474ba6","Type":"ContainerDied","Data":"f6f1afdb996d96eaa5ebaab0cca083c626c35eede28d77574a289826f69a8d9b"}
Oct 29 22:46:00.451159 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:00.451137    5428 generic.go:356] "Generic (PLEG): container finished" podID="fb43c41a-587f-47d6-affe-3231ce614d8d" containerID="faba27e3be2438df4891673f5876e50166bfe2159090bfae8cd92214c50e4ce4" exitCode=0
Oct 29 22:46:00.451249 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:00.451218    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59" event={"ID":"fb43c41a-587f-47d6-affe-3231ce614d8d","Type":"ContainerDied","Data":"faba27e3be2438df4891673f5876e50166bfe2159090bfae8cd92214c50e4ce4"}
Oct 29 22:46:00.453972 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:00.453341    5428 generic.go:356] "Generic (PLEG): container finished" podID="44c71a21-a3e3-4344-b32b-b39b09a7a440" containerID="94c8c06732bb87f02cf742633a38094e3ff731d52f85367b6d3f3c3eafb056c8" exitCode=0
Oct 29 22:46:00.453972 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:00.453396    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8" event={"ID":"44c71a21-a3e3-4344-b32b-b39b09a7a440","Type":"ContainerDied","Data":"94c8c06732bb87f02cf742633a38094e3ff731d52f85367b6d3f3c3eafb056c8"}
Oct 29 22:46:00.792856 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:00.792778    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:00.905752 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:00.905717    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:01.460688 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:01.460662    5428 generic.go:356] "Generic (PLEG): container finished" podID="edb57588-cd4b-4881-8976-d45d43474ba6" containerID="1b4ea5141d5d9bab6d0962a25cd4fddd5dcdaf4bac05e5d04295c3bff2469bb5" exitCode=0
Oct 29 22:46:01.461102 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:01.460748    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx" event={"ID":"edb57588-cd4b-4881-8976-d45d43474ba6","Type":"ContainerDied","Data":"1b4ea5141d5d9bab6d0962a25cd4fddd5dcdaf4bac05e5d04295c3bff2469bb5"}
Oct 29 22:46:01.463824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:01.463717    5428 generic.go:356] "Generic (PLEG): container finished" podID="fb43c41a-587f-47d6-affe-3231ce614d8d" containerID="a01f2971e49a99cd7839084a74d89aff38963efd7e5ded530f560d19a4b2d88e" exitCode=0
Oct 29 22:46:01.463824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:01.463790    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59" event={"ID":"fb43c41a-587f-47d6-affe-3231ce614d8d","Type":"ContainerDied","Data":"a01f2971e49a99cd7839084a74d89aff38963efd7e5ded530f560d19a4b2d88e"}
Oct 29 22:46:01.468695 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:01.468668    5428 generic.go:356] "Generic (PLEG): container finished" podID="44c71a21-a3e3-4344-b32b-b39b09a7a440" containerID="ff30d267832d9df9b045e05882f55207fe6f2a1ff09eb6983d2c3180674e6b57" exitCode=0
Oct 29 22:46:01.468791 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:01.468748    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8" event={"ID":"44c71a21-a3e3-4344-b32b-b39b09a7a440","Type":"ContainerDied","Data":"ff30d267832d9df9b045e05882f55207fe6f2a1ff09eb6983d2c3180674e6b57"}
Oct 29 22:46:01.472420 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:01.472395    5428 generic.go:356] "Generic (PLEG): container finished" podID="ae7e30b7-96de-442d-bd5b-149161c4df5b" containerID="2ab0188903accf4cd1d3f2e448c7144f4209826d65e9d1ce1e8a5290ab854783" exitCode=0
Oct 29 22:46:01.472517 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:01.472458    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d" event={"ID":"ae7e30b7-96de-442d-bd5b-149161c4df5b","Type":"ContainerDied","Data":"2ab0188903accf4cd1d3f2e448c7144f4209826d65e9d1ce1e8a5290ab854783"}
Oct 29 22:46:02.482164 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:02.481145    5428 generic.go:356] "Generic (PLEG): container finished" podID="133574bb-7b45-4950-b58b-758c5fff1df1" containerID="a0920ebaf99389c188ea4112af27969b7dbdbfcc4f825e84d49656901fa448e2" exitCode=0
Oct 29 22:46:02.482164 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:02.481307    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb" event={"ID":"133574bb-7b45-4950-b58b-758c5fff1df1","Type":"ContainerDied","Data":"a0920ebaf99389c188ea4112af27969b7dbdbfcc4f825e84d49656901fa448e2"}
Oct 29 22:46:02.486912 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:02.486282    5428 generic.go:356] "Generic (PLEG): container finished" podID="958235d6-aaec-45bf-a478-4ac4e20a272e" containerID="ff184ee11e55d74e6b62198fef83d32ff7b85001819aa3850974724b7e1297e1" exitCode=0
Oct 29 22:46:02.486912 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:02.486318    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g" event={"ID":"958235d6-aaec-45bf-a478-4ac4e20a272e","Type":"ContainerDied","Data":"ff184ee11e55d74e6b62198fef83d32ff7b85001819aa3850974724b7e1297e1"}
Oct 29 22:46:08.523900 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:08.523668    5428 generic.go:356] "Generic (PLEG): container finished" podID="958235d6-aaec-45bf-a478-4ac4e20a272e" containerID="a68fa19a9d042d43af30e7c4a7690629607d9df1808c20390bff52d0151175cd" exitCode=0
Oct 29 22:46:08.523900 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:08.523769    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g" event={"ID":"958235d6-aaec-45bf-a478-4ac4e20a272e","Type":"ContainerDied","Data":"a68fa19a9d042d43af30e7c4a7690629607d9df1808c20390bff52d0151175cd"}
Oct 29 22:46:08.527114 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:08.527010    5428 generic.go:356] "Generic (PLEG): container finished" podID="133574bb-7b45-4950-b58b-758c5fff1df1" containerID="1ccc9c8202a894de8d3552f77f5fb57fcb9db25beb281676ef52919d6a0b26ee" exitCode=0
Oct 29 22:46:08.527114 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:08.527075    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb" event={"ID":"133574bb-7b45-4950-b58b-758c5fff1df1","Type":"ContainerDied","Data":"1ccc9c8202a894de8d3552f77f5fb57fcb9db25beb281676ef52919d6a0b26ee"}
Oct 29 22:46:14.587778 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:14.586919    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc" event={"ID":"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40","Type":"ContainerStarted","Data":"e9606b093399185bb82a6bb2d49396d8b51d1c53a17959d856e29642c4ccd53d"}
Oct 29 22:46:14.628995 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:14.628345    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc" podStartSLOduration=2.186558153 podStartE2EDuration="30.628327704s" totalImagesPullingTime="28.441769551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:14.62762152 +0000 UTC m=+177.970570269" watchObservedRunningTime="2026-10-29 22:46:14.628327704 +0000 UTC m=+177.971276450"
Oct 29 22:46:15.594191 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:15.594156    5428 generic.go:356] "Generic (PLEG): container finished" podID="5ea52eb5-27d8-4ae6-adac-80fc3efb3f40" containerID="e9606b093399185bb82a6bb2d49396d8b51d1c53a17959d856e29642c4ccd53d" exitCode=0
Oct 29 22:46:15.594573 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:15.594242    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc" event={"ID":"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40","Type":"ContainerDied","Data":"e9606b093399185bb82a6bb2d49396d8b51d1c53a17959d856e29642c4ccd53d"}
Oct 29 22:46:15.597063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:15.597027    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx" event={"ID":"edb57588-cd4b-4881-8976-d45d43474ba6","Type":"ContainerStarted","Data":"1ee5b9947a749dcb15667fbd11bdb5dcbb5a761a55b4bb228017dad926ce8482"}
Oct 29 22:46:16.603859 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.603832    5428 generic.go:356] "Generic (PLEG): container finished" podID="44c71a21-a3e3-4344-b32b-b39b09a7a440" containerID="bf8eb494d2925bfca1926afa64e5e17109e0cf02e9a15a9077a0e38c8accd06b" exitCode=0
Oct 29 22:46:16.604240 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.603926    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8" event={"ID":"44c71a21-a3e3-4344-b32b-b39b09a7a440","Type":"ContainerDied","Data":"bf8eb494d2925bfca1926afa64e5e17109e0cf02e9a15a9077a0e38c8accd06b"}
Oct 29 22:46:16.605910 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.605882    5428 generic.go:356] "Generic (PLEG): container finished" podID="958235d6-aaec-45bf-a478-4ac4e20a272e" containerID="b9fe9e886a0d7d3d9680b652341d2564bce2ad98061c799b2920926388981ee9" exitCode=0
Oct 29 22:46:16.606031 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.605959    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g" event={"ID":"958235d6-aaec-45bf-a478-4ac4e20a272e","Type":"ContainerDied","Data":"b9fe9e886a0d7d3d9680b652341d2564bce2ad98061c799b2920926388981ee9"}
Oct 29 22:46:16.608092 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.608063    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb" event={"ID":"133574bb-7b45-4950-b58b-758c5fff1df1","Type":"ContainerStarted","Data":"540a2ca9be66e7c20c911bcee6e0608f252a8481c73a982fcdeb92e0c08faede"}
Oct 29 22:46:16.610424 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.610404    5428 generic.go:356] "Generic (PLEG): container finished" podID="ae7e30b7-96de-442d-bd5b-149161c4df5b" containerID="9256b989211837dc91a265e323194a99ce2b2ca691153f6aea1155e741280f3d" exitCode=0
Oct 29 22:46:16.610511 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.610486    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d" event={"ID":"ae7e30b7-96de-442d-bd5b-149161c4df5b","Type":"ContainerDied","Data":"9256b989211837dc91a265e323194a99ce2b2ca691153f6aea1155e741280f3d"}
Oct 29 22:46:16.612440 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.612420    5428 generic.go:356] "Generic (PLEG): container finished" podID="edb57588-cd4b-4881-8976-d45d43474ba6" containerID="1ee5b9947a749dcb15667fbd11bdb5dcbb5a761a55b4bb228017dad926ce8482" exitCode=0
Oct 29 22:46:16.612610 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.612494    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx" event={"ID":"edb57588-cd4b-4881-8976-d45d43474ba6","Type":"ContainerDied","Data":"1ee5b9947a749dcb15667fbd11bdb5dcbb5a761a55b4bb228017dad926ce8482"}
Oct 29 22:46:16.614362 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.614340    5428 generic.go:356] "Generic (PLEG): container finished" podID="fb43c41a-587f-47d6-affe-3231ce614d8d" containerID="e20639554ccbbcbe5831bd702b2650725102d9a646da34c8988d92766a6b8de2" exitCode=0
Oct 29 22:46:16.614439 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.614412    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59" event={"ID":"fb43c41a-587f-47d6-affe-3231ce614d8d","Type":"ContainerDied","Data":"e20639554ccbbcbe5831bd702b2650725102d9a646da34c8988d92766a6b8de2"}
Oct 29 22:46:16.724108 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.724064    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb" podStartSLOduration=2.906228266 podStartE2EDuration="28.724050488s" totalImagesPullingTime="25.817822222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:16.723362549 +0000 UTC m=+180.066311293" watchObservedRunningTime="2026-10-29 22:46:16.724050488 +0000 UTC m=+180.066999233"
Oct 29 22:46:16.739591 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.739570    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc"
Oct 29 22:46:16.886609 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.886576    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-kube-api-access-hd86q\" (UniqueName: \"kubernetes.io/projected/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-kube-api-access-hd86q\") pod \"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40\" (UID: \"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40\") "
Oct 29 22:46:16.886791 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.886684    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-bundle\") pod \"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40\" (UID: \"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40\") "
Oct 29 22:46:16.886791 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.886732    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-util\" (UniqueName: \"kubernetes.io/empty-dir/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-util\") pod \"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40\" (UID: \"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40\") "
Oct 29 22:46:16.887370 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.887327    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-bundle" pod "5ea52eb5-27d8-4ae6-adac-80fc3efb3f40" (UID: "5ea52eb5-27d8-4ae6-adac-80fc3efb3f40"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:16.888987 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.888953    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-kube-api-access-hd86q" pod "5ea52eb5-27d8-4ae6-adac-80fc3efb3f40" (UID: "5ea52eb5-27d8-4ae6-adac-80fc3efb3f40"). InnerVolumeSpecName "kube-api-access-hd86q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:16.892896 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.892873    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-util" pod "5ea52eb5-27d8-4ae6-adac-80fc3efb3f40" (UID: "5ea52eb5-27d8-4ae6-adac-80fc3efb3f40"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:16.987627 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.987597    5428 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-bundle\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:16.987627 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.987621    5428 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-util\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:16.987627 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:16.987632    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd86q\" (UniqueName: \"kubernetes.io/projected/5ea52eb5-27d8-4ae6-adac-80fc3efb3f40-kube-api-access-hd86q\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:17.365150 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:46:17.365111    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:46:17.619894 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.619816    5428 generic.go:356] "Generic (PLEG): container finished" podID="133574bb-7b45-4950-b58b-758c5fff1df1" containerID="540a2ca9be66e7c20c911bcee6e0608f252a8481c73a982fcdeb92e0c08faede" exitCode=0
Oct 29 22:46:17.620315 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.619897    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb" event={"ID":"133574bb-7b45-4950-b58b-758c5fff1df1","Type":"ContainerDied","Data":"540a2ca9be66e7c20c911bcee6e0608f252a8481c73a982fcdeb92e0c08faede"}
Oct 29 22:46:17.621765 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.621743    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc"
Oct 29 22:46:17.621887 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.621765    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719x2pc" event={"ID":"5ea52eb5-27d8-4ae6-adac-80fc3efb3f40","Type":"ContainerDied","Data":"1ad4fcc665dc89b116eacab5a7061383402ff5fc228b8456a9ba166a3696341a"}
Oct 29 22:46:17.621887 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.621807    5428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ad4fcc665dc89b116eacab5a7061383402ff5fc228b8456a9ba166a3696341a"
Oct 29 22:46:17.762252 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.762227    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59"
Oct 29 22:46:17.839692 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.839663    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d"
Oct 29 22:46:17.845968 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.845949    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx"
Oct 29 22:46:17.849175 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.849156    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8"
Oct 29 22:46:17.852377 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.852359    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g"
Oct 29 22:46:17.896698 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.896645    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb43c41a-587f-47d6-affe-3231ce614d8d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb43c41a-587f-47d6-affe-3231ce614d8d-bundle\") pod \"fb43c41a-587f-47d6-affe-3231ce614d8d\" (UID: \"fb43c41a-587f-47d6-affe-3231ce614d8d\") "
Oct 29 22:46:17.896792 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.896722    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb43c41a-587f-47d6-affe-3231ce614d8d-kube-api-access-cx75t\" (UniqueName: \"kubernetes.io/projected/fb43c41a-587f-47d6-affe-3231ce614d8d-kube-api-access-cx75t\") pod \"fb43c41a-587f-47d6-affe-3231ce614d8d\" (UID: \"fb43c41a-587f-47d6-affe-3231ce614d8d\") "
Oct 29 22:46:17.896792 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.896761    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb43c41a-587f-47d6-affe-3231ce614d8d-util\" (UniqueName: \"kubernetes.io/empty-dir/fb43c41a-587f-47d6-affe-3231ce614d8d-util\") pod \"fb43c41a-587f-47d6-affe-3231ce614d8d\" (UID: \"fb43c41a-587f-47d6-affe-3231ce614d8d\") "
Oct 29 22:46:17.897290 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.897264    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb43c41a-587f-47d6-affe-3231ce614d8d-bundle" pod "fb43c41a-587f-47d6-affe-3231ce614d8d" (UID: "fb43c41a-587f-47d6-affe-3231ce614d8d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:17.898748 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.898727    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb43c41a-587f-47d6-affe-3231ce614d8d-kube-api-access-cx75t" pod "fb43c41a-587f-47d6-affe-3231ce614d8d" (UID: "fb43c41a-587f-47d6-affe-3231ce614d8d"). InnerVolumeSpecName "kube-api-access-cx75t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:17.902781 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.902761    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb43c41a-587f-47d6-affe-3231ce614d8d-util" pod "fb43c41a-587f-47d6-affe-3231ce614d8d" (UID: "fb43c41a-587f-47d6-affe-3231ce614d8d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:17.998112 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.998084    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7e30b7-96de-442d-bd5b-149161c4df5b-kube-api-access-86n8x\" (UniqueName: \"kubernetes.io/projected/ae7e30b7-96de-442d-bd5b-149161c4df5b-kube-api-access-86n8x\") pod \"ae7e30b7-96de-442d-bd5b-149161c4df5b\" (UID: \"ae7e30b7-96de-442d-bd5b-149161c4df5b\") "
Oct 29 22:46:17.998287 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.998120    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/958235d6-aaec-45bf-a478-4ac4e20a272e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/958235d6-aaec-45bf-a478-4ac4e20a272e-bundle\") pod \"958235d6-aaec-45bf-a478-4ac4e20a272e\" (UID: \"958235d6-aaec-45bf-a478-4ac4e20a272e\") "
Oct 29 22:46:17.998287 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.998153    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c71a21-a3e3-4344-b32b-b39b09a7a440-kube-api-access-nsbwk\" (UniqueName: \"kubernetes.io/projected/44c71a21-a3e3-4344-b32b-b39b09a7a440-kube-api-access-nsbwk\") pod \"44c71a21-a3e3-4344-b32b-b39b09a7a440\" (UID: \"44c71a21-a3e3-4344-b32b-b39b09a7a440\") "
Oct 29 22:46:17.998287 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.998173    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c71a21-a3e3-4344-b32b-b39b09a7a440-bundle\" (UniqueName: \"kubernetes.io/empty-dir/44c71a21-a3e3-4344-b32b-b39b09a7a440-bundle\") pod \"44c71a21-a3e3-4344-b32b-b39b09a7a440\" (UID: \"44c71a21-a3e3-4344-b32b-b39b09a7a440\") "
Oct 29 22:46:17.998435 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.998286    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/958235d6-aaec-45bf-a478-4ac4e20a272e-kube-api-access-9r87j\" (UniqueName: \"kubernetes.io/projected/958235d6-aaec-45bf-a478-4ac4e20a272e-kube-api-access-9r87j\") pod \"958235d6-aaec-45bf-a478-4ac4e20a272e\" (UID: \"958235d6-aaec-45bf-a478-4ac4e20a272e\") "
Oct 29 22:46:17.998435 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.998320    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7e30b7-96de-442d-bd5b-149161c4df5b-util\" (UniqueName: \"kubernetes.io/empty-dir/ae7e30b7-96de-442d-bd5b-149161c4df5b-util\") pod \"ae7e30b7-96de-442d-bd5b-149161c4df5b\" (UID: \"ae7e30b7-96de-442d-bd5b-149161c4df5b\") "
Oct 29 22:46:17.998435 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.998363    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edb57588-cd4b-4881-8976-d45d43474ba6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/edb57588-cd4b-4881-8976-d45d43474ba6-bundle\") pod \"edb57588-cd4b-4881-8976-d45d43474ba6\" (UID: \"edb57588-cd4b-4881-8976-d45d43474ba6\") "
Oct 29 22:46:17.998435 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.998426    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/958235d6-aaec-45bf-a478-4ac4e20a272e-util\" (UniqueName: \"kubernetes.io/empty-dir/958235d6-aaec-45bf-a478-4ac4e20a272e-util\") pod \"958235d6-aaec-45bf-a478-4ac4e20a272e\" (UID: \"958235d6-aaec-45bf-a478-4ac4e20a272e\") "
Oct 29 22:46:17.998634 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.998462    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edb57588-cd4b-4881-8976-d45d43474ba6-kube-api-access-2kvx7\" (UniqueName: \"kubernetes.io/projected/edb57588-cd4b-4881-8976-d45d43474ba6-kube-api-access-2kvx7\") pod \"edb57588-cd4b-4881-8976-d45d43474ba6\" (UID: \"edb57588-cd4b-4881-8976-d45d43474ba6\") "
Oct 29 22:46:17.998634 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.998497    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44c71a21-a3e3-4344-b32b-b39b09a7a440-util\" (UniqueName: \"kubernetes.io/empty-dir/44c71a21-a3e3-4344-b32b-b39b09a7a440-util\") pod \"44c71a21-a3e3-4344-b32b-b39b09a7a440\" (UID: \"44c71a21-a3e3-4344-b32b-b39b09a7a440\") "
Oct 29 22:46:17.998634 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.998521    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edb57588-cd4b-4881-8976-d45d43474ba6-util\" (UniqueName: \"kubernetes.io/empty-dir/edb57588-cd4b-4881-8976-d45d43474ba6-util\") pod \"edb57588-cd4b-4881-8976-d45d43474ba6\" (UID: \"edb57588-cd4b-4881-8976-d45d43474ba6\") "
Oct 29 22:46:17.998634 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.998566    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7e30b7-96de-442d-bd5b-149161c4df5b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae7e30b7-96de-442d-bd5b-149161c4df5b-bundle\") pod \"ae7e30b7-96de-442d-bd5b-149161c4df5b\" (UID: \"ae7e30b7-96de-442d-bd5b-149161c4df5b\") "
Oct 29 22:46:17.998874 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.998681    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958235d6-aaec-45bf-a478-4ac4e20a272e-bundle" pod "958235d6-aaec-45bf-a478-4ac4e20a272e" (UID: "958235d6-aaec-45bf-a478-4ac4e20a272e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:17.998874 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.998845    5428 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/958235d6-aaec-45bf-a478-4ac4e20a272e-bundle\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:17.998874 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.998864    5428 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb43c41a-587f-47d6-affe-3231ce614d8d-bundle\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:17.999929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.998878    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx75t\" (UniqueName: \"kubernetes.io/projected/fb43c41a-587f-47d6-affe-3231ce614d8d-kube-api-access-cx75t\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:17.999929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.998891    5428 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb43c41a-587f-47d6-affe-3231ce614d8d-util\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:17.999929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.999354    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c71a21-a3e3-4344-b32b-b39b09a7a440-bundle" pod "44c71a21-a3e3-4344-b32b-b39b09a7a440" (UID: "44c71a21-a3e3-4344-b32b-b39b09a7a440"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:17.999929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.999385    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb57588-cd4b-4881-8976-d45d43474ba6-bundle" pod "edb57588-cd4b-4881-8976-d45d43474ba6" (UID: "edb57588-cd4b-4881-8976-d45d43474ba6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:17.999929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:17.999779    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7e30b7-96de-442d-bd5b-149161c4df5b-bundle" pod "ae7e30b7-96de-442d-bd5b-149161c4df5b" (UID: "ae7e30b7-96de-442d-bd5b-149161c4df5b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:18.001216 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.001182    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c71a21-a3e3-4344-b32b-b39b09a7a440-kube-api-access-nsbwk" pod "44c71a21-a3e3-4344-b32b-b39b09a7a440" (UID: "44c71a21-a3e3-4344-b32b-b39b09a7a440"). InnerVolumeSpecName "kube-api-access-nsbwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:18.001470 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.001447    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958235d6-aaec-45bf-a478-4ac4e20a272e-kube-api-access-9r87j" pod "958235d6-aaec-45bf-a478-4ac4e20a272e" (UID: "958235d6-aaec-45bf-a478-4ac4e20a272e"). InnerVolumeSpecName "kube-api-access-9r87j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:18.001921 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.001871    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb57588-cd4b-4881-8976-d45d43474ba6-kube-api-access-2kvx7" pod "edb57588-cd4b-4881-8976-d45d43474ba6" (UID: "edb57588-cd4b-4881-8976-d45d43474ba6"). InnerVolumeSpecName "kube-api-access-2kvx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:18.002182 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.002147    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7e30b7-96de-442d-bd5b-149161c4df5b-kube-api-access-86n8x" pod "ae7e30b7-96de-442d-bd5b-149161c4df5b" (UID: "ae7e30b7-96de-442d-bd5b-149161c4df5b"). InnerVolumeSpecName "kube-api-access-86n8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:18.006032 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.006003    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7e30b7-96de-442d-bd5b-149161c4df5b-util" pod "ae7e30b7-96de-442d-bd5b-149161c4df5b" (UID: "ae7e30b7-96de-442d-bd5b-149161c4df5b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:18.007012 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.006991    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c71a21-a3e3-4344-b32b-b39b09a7a440-util" pod "44c71a21-a3e3-4344-b32b-b39b09a7a440" (UID: "44c71a21-a3e3-4344-b32b-b39b09a7a440"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:18.007353 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.007329    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb57588-cd4b-4881-8976-d45d43474ba6-util" pod "edb57588-cd4b-4881-8976-d45d43474ba6" (UID: "edb57588-cd4b-4881-8976-d45d43474ba6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:18.007539 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.007521    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958235d6-aaec-45bf-a478-4ac4e20a272e-util" pod "958235d6-aaec-45bf-a478-4ac4e20a272e" (UID: "958235d6-aaec-45bf-a478-4ac4e20a272e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:18.099711 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.099676    5428 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/958235d6-aaec-45bf-a478-4ac4e20a272e-util\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.099711 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.099704    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kvx7\" (UniqueName: \"kubernetes.io/projected/edb57588-cd4b-4881-8976-d45d43474ba6-kube-api-access-2kvx7\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.099711 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.099713    5428 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44c71a21-a3e3-4344-b32b-b39b09a7a440-util\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.099972 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.099722    5428 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edb57588-cd4b-4881-8976-d45d43474ba6-util\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.099972 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.099734    5428 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae7e30b7-96de-442d-bd5b-149161c4df5b-bundle\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.099972 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.099742    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86n8x\" (UniqueName: \"kubernetes.io/projected/ae7e30b7-96de-442d-bd5b-149161c4df5b-kube-api-access-86n8x\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.099972 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.099750    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsbwk\" (UniqueName: \"kubernetes.io/projected/44c71a21-a3e3-4344-b32b-b39b09a7a440-kube-api-access-nsbwk\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.099972 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.099760    5428 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44c71a21-a3e3-4344-b32b-b39b09a7a440-bundle\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.099972 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.099769    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r87j\" (UniqueName: \"kubernetes.io/projected/958235d6-aaec-45bf-a478-4ac4e20a272e-kube-api-access-9r87j\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.099972 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.099776    5428 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae7e30b7-96de-442d-bd5b-149161c4df5b-util\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.099972 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.099786    5428 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edb57588-cd4b-4881-8976-d45d43474ba6-bundle\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.627069 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.627039    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx"
Oct 29 22:46:18.627504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.627034    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rmx" event={"ID":"edb57588-cd4b-4881-8976-d45d43474ba6","Type":"ContainerDied","Data":"b4f8b4973489fa881813e025579c903c2647a6f4990c19d00e895f3b3e9f8960"}
Oct 29 22:46:18.627504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.627167    5428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4f8b4973489fa881813e025579c903c2647a6f4990c19d00e895f3b3e9f8960"
Oct 29 22:46:18.629025 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.629007    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59"
Oct 29 22:46:18.629025 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.629015    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfcv59" event={"ID":"fb43c41a-587f-47d6-affe-3231ce614d8d","Type":"ContainerDied","Data":"d640da4dd36938d09a3112653094321fbc2f23907fde43a9fd86c34d3d71e150"}
Oct 29 22:46:18.629177 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.629054    5428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d640da4dd36938d09a3112653094321fbc2f23907fde43a9fd86c34d3d71e150"
Oct 29 22:46:18.631127 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.631111    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8"
Oct 29 22:46:18.631234 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.631148    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av8tb8" event={"ID":"44c71a21-a3e3-4344-b32b-b39b09a7a440","Type":"ContainerDied","Data":"d751865b6d753e1291a2b784ffd2a23f2b097ce5dea4f68dbe2a9b70abbe1c59"}
Oct 29 22:46:18.631234 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.631185    5428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d751865b6d753e1291a2b784ffd2a23f2b097ce5dea4f68dbe2a9b70abbe1c59"
Oct 29 22:46:18.633337 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.633320    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g"
Oct 29 22:46:18.633420 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.633334    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xct8g" event={"ID":"958235d6-aaec-45bf-a478-4ac4e20a272e","Type":"ContainerDied","Data":"42d39dddaf64bdd45c802cdf06d1e099b80ba37938891d76185debfab0d83a82"}
Oct 29 22:46:18.633420 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.633405    5428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42d39dddaf64bdd45c802cdf06d1e099b80ba37938891d76185debfab0d83a82"
Oct 29 22:46:18.635305 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.635285    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d"
Oct 29 22:46:18.635380 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.635331    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jl9d" event={"ID":"ae7e30b7-96de-442d-bd5b-149161c4df5b","Type":"ContainerDied","Data":"01982a0fbbe250dc328109d35039cbd89302454b631638c1af79dc6d0247f09c"}
Oct 29 22:46:18.635380 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.635361    5428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01982a0fbbe250dc328109d35039cbd89302454b631638c1af79dc6d0247f09c"
Oct 29 22:46:18.762447 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.762418    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb"
Oct 29 22:46:18.905926 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.905892    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133574bb-7b45-4950-b58b-758c5fff1df1-util\" (UniqueName: \"kubernetes.io/empty-dir/133574bb-7b45-4950-b58b-758c5fff1df1-util\") pod \"133574bb-7b45-4950-b58b-758c5fff1df1\" (UID: \"133574bb-7b45-4950-b58b-758c5fff1df1\") "
Oct 29 22:46:18.906086 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.905951    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133574bb-7b45-4950-b58b-758c5fff1df1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/133574bb-7b45-4950-b58b-758c5fff1df1-bundle\") pod \"133574bb-7b45-4950-b58b-758c5fff1df1\" (UID: \"133574bb-7b45-4950-b58b-758c5fff1df1\") "
Oct 29 22:46:18.906086 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.906056    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133574bb-7b45-4950-b58b-758c5fff1df1-kube-api-access-h42dv\" (UniqueName: \"kubernetes.io/projected/133574bb-7b45-4950-b58b-758c5fff1df1-kube-api-access-h42dv\") pod \"133574bb-7b45-4950-b58b-758c5fff1df1\" (UID: \"133574bb-7b45-4950-b58b-758c5fff1df1\") "
Oct 29 22:46:18.906862 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.906831    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133574bb-7b45-4950-b58b-758c5fff1df1-bundle" pod "133574bb-7b45-4950-b58b-758c5fff1df1" (UID: "133574bb-7b45-4950-b58b-758c5fff1df1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:18.908347 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.908321    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133574bb-7b45-4950-b58b-758c5fff1df1-kube-api-access-h42dv" pod "133574bb-7b45-4950-b58b-758c5fff1df1" (UID: "133574bb-7b45-4950-b58b-758c5fff1df1"). InnerVolumeSpecName "kube-api-access-h42dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:18.910228 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:18.910201    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133574bb-7b45-4950-b58b-758c5fff1df1-util" pod "133574bb-7b45-4950-b58b-758c5fff1df1" (UID: "133574bb-7b45-4950-b58b-758c5fff1df1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:19.007291 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:19.007255    5428 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/133574bb-7b45-4950-b58b-758c5fff1df1-util\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:19.007291 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:19.007290    5428 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/133574bb-7b45-4950-b58b-758c5fff1df1-bundle\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:19.007466 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:19.007306    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h42dv\" (UniqueName: \"kubernetes.io/projected/133574bb-7b45-4950-b58b-758c5fff1df1-kube-api-access-h42dv\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:46:19.641386 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:19.641359    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb"
Oct 29 22:46:19.641758 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:19.641358    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbv5jb" event={"ID":"133574bb-7b45-4950-b58b-758c5fff1df1","Type":"ContainerDied","Data":"f7482bdc0e7440cd200ae42393ecc043b1c7b720e1c15bf7cbc2aa41731e65b5"}
Oct 29 22:46:19.641758 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:19.641476    5428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7482bdc0e7440cd200ae42393ecc043b1c7b720e1c15bf7cbc2aa41731e65b5"
Oct 29 22:46:20.871045 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:20.871002    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:21.652493 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:21.652457    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:22.674313 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:22.674281    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:22.982037 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:22.981934    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:26.753423 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753391    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx"]
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753638    5428 cpu_manager.go:414] "containerMap: removing container" podUID="5ea52eb5-27d8-4ae6-adac-80fc3efb3f40" containerName="util"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753648    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea52eb5-27d8-4ae6-adac-80fc3efb3f40" containerName="util"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753658    5428 cpu_manager.go:414] "containerMap: removing container" podUID="44c71a21-a3e3-4344-b32b-b39b09a7a440" containerName="pull"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753663    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c71a21-a3e3-4344-b32b-b39b09a7a440" containerName="pull"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753670    5428 cpu_manager.go:414] "containerMap: removing container" podUID="133574bb-7b45-4950-b58b-758c5fff1df1" containerName="util"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753676    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="133574bb-7b45-4950-b58b-758c5fff1df1" containerName="util"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753682    5428 cpu_manager.go:414] "containerMap: removing container" podUID="5ea52eb5-27d8-4ae6-adac-80fc3efb3f40" containerName="extract"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753686    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea52eb5-27d8-4ae6-adac-80fc3efb3f40" containerName="extract"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753693    5428 cpu_manager.go:414] "containerMap: removing container" podUID="edb57588-cd4b-4881-8976-d45d43474ba6" containerName="extract"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753698    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb57588-cd4b-4881-8976-d45d43474ba6" containerName="extract"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753706    5428 cpu_manager.go:414] "containerMap: removing container" podUID="133574bb-7b45-4950-b58b-758c5fff1df1" containerName="pull"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753711    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="133574bb-7b45-4950-b58b-758c5fff1df1" containerName="pull"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753718    5428 cpu_manager.go:414] "containerMap: removing container" podUID="fb43c41a-587f-47d6-affe-3231ce614d8d" containerName="pull"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753723    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb43c41a-587f-47d6-affe-3231ce614d8d" containerName="pull"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753729    5428 cpu_manager.go:414] "containerMap: removing container" podUID="fb43c41a-587f-47d6-affe-3231ce614d8d" containerName="util"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753733    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb43c41a-587f-47d6-affe-3231ce614d8d" containerName="util"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753743    5428 cpu_manager.go:414] "containerMap: removing container" podUID="edb57588-cd4b-4881-8976-d45d43474ba6" containerName="util"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753748    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb57588-cd4b-4881-8976-d45d43474ba6" containerName="util"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753754    5428 cpu_manager.go:414] "containerMap: removing container" podUID="fb43c41a-587f-47d6-affe-3231ce614d8d" containerName="extract"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753758    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb43c41a-587f-47d6-affe-3231ce614d8d" containerName="extract"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753764    5428 cpu_manager.go:414] "containerMap: removing container" podUID="ae7e30b7-96de-442d-bd5b-149161c4df5b" containerName="extract"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753769    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7e30b7-96de-442d-bd5b-149161c4df5b" containerName="extract"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753778    5428 cpu_manager.go:414] "containerMap: removing container" podUID="44c71a21-a3e3-4344-b32b-b39b09a7a440" containerName="extract"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753782    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c71a21-a3e3-4344-b32b-b39b09a7a440" containerName="extract"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753790    5428 cpu_manager.go:414] "containerMap: removing container" podUID="958235d6-aaec-45bf-a478-4ac4e20a272e" containerName="util"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753795    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="958235d6-aaec-45bf-a478-4ac4e20a272e" containerName="util"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753815    5428 cpu_manager.go:414] "containerMap: removing container" podUID="5ea52eb5-27d8-4ae6-adac-80fc3efb3f40" containerName="pull"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753819    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea52eb5-27d8-4ae6-adac-80fc3efb3f40" containerName="pull"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753825    5428 cpu_manager.go:414] "containerMap: removing container" podUID="958235d6-aaec-45bf-a478-4ac4e20a272e" containerName="extract"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753829    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="958235d6-aaec-45bf-a478-4ac4e20a272e" containerName="extract"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753834    5428 cpu_manager.go:414] "containerMap: removing container" podUID="133574bb-7b45-4950-b58b-758c5fff1df1" containerName="extract"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753839    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="133574bb-7b45-4950-b58b-758c5fff1df1" containerName="extract"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753845    5428 cpu_manager.go:414] "containerMap: removing container" podUID="ae7e30b7-96de-442d-bd5b-149161c4df5b" containerName="util"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753849    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7e30b7-96de-442d-bd5b-149161c4df5b" containerName="util"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753854    5428 cpu_manager.go:414] "containerMap: removing container" podUID="958235d6-aaec-45bf-a478-4ac4e20a272e" containerName="pull"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753858    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="958235d6-aaec-45bf-a478-4ac4e20a272e" containerName="pull"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753867    5428 cpu_manager.go:414] "containerMap: removing container" podUID="44c71a21-a3e3-4344-b32b-b39b09a7a440" containerName="util"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753871    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c71a21-a3e3-4344-b32b-b39b09a7a440" containerName="util"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753879    5428 cpu_manager.go:414] "containerMap: removing container" podUID="ae7e30b7-96de-442d-bd5b-149161c4df5b" containerName="pull"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753883    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7e30b7-96de-442d-bd5b-149161c4df5b" containerName="pull"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753889    5428 cpu_manager.go:414] "containerMap: removing container" podUID="edb57588-cd4b-4881-8976-d45d43474ba6" containerName="pull"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753893    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb57588-cd4b-4881-8976-d45d43474ba6" containerName="pull"
Oct 29 22:46:26.753929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753932    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb43c41a-587f-47d6-affe-3231ce614d8d" containerName="extract"
Oct 29 22:46:26.755386 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753941    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae7e30b7-96de-442d-bd5b-149161c4df5b" containerName="extract"
Oct 29 22:46:26.755386 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753962    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c71a21-a3e3-4344-b32b-b39b09a7a440" containerName="extract"
Oct 29 22:46:26.755386 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753968    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="958235d6-aaec-45bf-a478-4ac4e20a272e" containerName="extract"
Oct 29 22:46:26.755386 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753973    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="133574bb-7b45-4950-b58b-758c5fff1df1" containerName="extract"
Oct 29 22:46:26.755386 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753980    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea52eb5-27d8-4ae6-adac-80fc3efb3f40" containerName="extract"
Oct 29 22:46:26.755386 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.753984    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb57588-cd4b-4881-8976-d45d43474ba6" containerName="extract"
Oct 29 22:46:26.756216 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.756200    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx"
Oct 29 22:46:26.758888 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.758864    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/21d3b0f9-5747-40c0-b610-0efc8e08a6ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-wtqwx\" (UID: \"21d3b0f9-5747-40c0-b610-0efc8e08a6ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx"
Oct 29 22:46:26.758999 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.758903    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfqc4\" (UniqueName: \"kubernetes.io/projected/21d3b0f9-5747-40c0-b610-0efc8e08a6ce-kube-api-access-jfqc4\") pod \"kuadrant-operator-controller-manager-cf8469d7b-wtqwx\" (UID: \"21d3b0f9-5747-40c0-b610-0efc8e08a6ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx"
Oct 29 22:46:26.771838 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.771814    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 29 22:46:26.771966 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.771952    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 29 22:46:26.782036 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.782013    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f8bng\""
Oct 29 22:46:26.783392 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.783365    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx"]
Oct 29 22:46:26.859721 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.859691    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/21d3b0f9-5747-40c0-b610-0efc8e08a6ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-wtqwx\" (UID: \"21d3b0f9-5747-40c0-b610-0efc8e08a6ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx"
Oct 29 22:46:26.859851 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.859731    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfqc4\" (UniqueName: \"kubernetes.io/projected/21d3b0f9-5747-40c0-b610-0efc8e08a6ce-kube-api-access-jfqc4\") pod \"kuadrant-operator-controller-manager-cf8469d7b-wtqwx\" (UID: \"21d3b0f9-5747-40c0-b610-0efc8e08a6ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx"
Oct 29 22:46:26.860094 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.860074    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/21d3b0f9-5747-40c0-b610-0efc8e08a6ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-wtqwx\" (UID: \"21d3b0f9-5747-40c0-b610-0efc8e08a6ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx"
Oct 29 22:46:26.884297 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:26.884263    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfqc4\" (UniqueName: \"kubernetes.io/projected/21d3b0f9-5747-40c0-b610-0efc8e08a6ce-kube-api-access-jfqc4\") pod \"kuadrant-operator-controller-manager-cf8469d7b-wtqwx\" (UID: \"21d3b0f9-5747-40c0-b610-0efc8e08a6ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx"
Oct 29 22:46:27.062316 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:27.062227    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:27.067081 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:27.067058    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx"
Oct 29 22:46:27.234553 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:27.234518    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:27.402435 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:27.402404    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx"]
Oct 29 22:46:27.407003 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:46:27.406979    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d3b0f9_5747_40c0_b610_0efc8e08a6ce.slice/crio-2782a5db87fdf01a03b1235ce0734cfef88864a7922fd2c816223c24033c327b WatchSource:0}: Error finding container 2782a5db87fdf01a03b1235ce0734cfef88864a7922fd2c816223c24033c327b: Status 404 returned error can't find the container with id 2782a5db87fdf01a03b1235ce0734cfef88864a7922fd2c816223c24033c327b
Oct 29 22:46:27.672405 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:27.672304    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx" event={"ID":"21d3b0f9-5747-40c0-b610-0efc8e08a6ce","Type":"ContainerStarted","Data":"2782a5db87fdf01a03b1235ce0734cfef88864a7922fd2c816223c24033c327b"}
Oct 29 22:46:28.316931 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.316877    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:28.452884 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.452840    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:28.545784 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.545742    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"]
Oct 29 22:46:28.549655 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.549624    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.572059 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.571973    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/050c87b1-f0de-47c6-82cd-2c2eac0e0c1f-metrics-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-882vr\" (UID: \"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.572059 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.572020    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk2t2\" (UniqueName: \"kubernetes.io/projected/050c87b1-f0de-47c6-82cd-2c2eac0e0c1f-kube-api-access-sk2t2\") pod \"opendatahub-operator-controller-manager-844ffc67f7-882vr\" (UID: \"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.572059 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.572055    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/050c87b1-f0de-47c6-82cd-2c2eac0e0c1f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-882vr\" (UID: \"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.572291 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.572141    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/050c87b1-f0de-47c6-82cd-2c2eac0e0c1f-tmp\") pod \"opendatahub-operator-controller-manager-844ffc67f7-882vr\" (UID: \"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.572291 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.572217    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/050c87b1-f0de-47c6-82cd-2c2eac0e0c1f-manifests\") pod \"opendatahub-operator-controller-manager-844ffc67f7-882vr\" (UID: \"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.572291 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.572246    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/050c87b1-f0de-47c6-82cd-2c2eac0e0c1f-webhook-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-882vr\" (UID: \"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.577554 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.577526    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 29 22:46:28.587758 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.587734    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 29 22:46:28.637094 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.637056    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"]
Oct 29 22:46:28.654227 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.654193    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 29 22:46:28.654396 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.654225    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-c59b2\""
Oct 29 22:46:28.654396 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.654261    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 29 22:46:28.656702 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.655932    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 29 22:46:28.673588 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.673561    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/050c87b1-f0de-47c6-82cd-2c2eac0e0c1f-metrics-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-882vr\" (UID: \"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.673756 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.673598    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk2t2\" (UniqueName: \"kubernetes.io/projected/050c87b1-f0de-47c6-82cd-2c2eac0e0c1f-kube-api-access-sk2t2\") pod \"opendatahub-operator-controller-manager-844ffc67f7-882vr\" (UID: \"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.673756 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.673634    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/050c87b1-f0de-47c6-82cd-2c2eac0e0c1f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-882vr\" (UID: \"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.673756 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.673655    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/050c87b1-f0de-47c6-82cd-2c2eac0e0c1f-tmp\") pod \"opendatahub-operator-controller-manager-844ffc67f7-882vr\" (UID: \"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.673756 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.673676    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/050c87b1-f0de-47c6-82cd-2c2eac0e0c1f-manifests\") pod \"opendatahub-operator-controller-manager-844ffc67f7-882vr\" (UID: \"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.673756 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.673693    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/050c87b1-f0de-47c6-82cd-2c2eac0e0c1f-webhook-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-882vr\" (UID: \"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.674418 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.674375    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/050c87b1-f0de-47c6-82cd-2c2eac0e0c1f-manifests\") pod \"opendatahub-operator-controller-manager-844ffc67f7-882vr\" (UID: \"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.676671 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.676649    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/050c87b1-f0de-47c6-82cd-2c2eac0e0c1f-tmp\") pod \"opendatahub-operator-controller-manager-844ffc67f7-882vr\" (UID: \"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.677203 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.677156    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/050c87b1-f0de-47c6-82cd-2c2eac0e0c1f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-882vr\" (UID: \"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.678917 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.677677    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/050c87b1-f0de-47c6-82cd-2c2eac0e0c1f-webhook-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-882vr\" (UID: \"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.678917 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.678630    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/050c87b1-f0de-47c6-82cd-2c2eac0e0c1f-metrics-cert\") pod \"opendatahub-operator-controller-manager-844ffc67f7-882vr\" (UID: \"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.696643 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.696621    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk2t2\" (UniqueName: \"kubernetes.io/projected/050c87b1-f0de-47c6-82cd-2c2eac0e0c1f-kube-api-access-sk2t2\") pod \"opendatahub-operator-controller-manager-844ffc67f7-882vr\" (UID: \"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.782047 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.782011    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:28.869980 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.869946    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:28.912664 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:28.912590    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:29.022515 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:29.022475    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:29.141030 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:29.140895    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:29.238269 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:29.238230    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"]
Oct 29 22:46:29.242728 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:46:29.242693    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050c87b1_f0de_47c6_82cd_2c2eac0e0c1f.slice/crio-b33e305f2b8b2e902c766c6b377cf6c9c4db4a8c5ea31a7d400b081095be7100 WatchSource:0}: Error finding container b33e305f2b8b2e902c766c6b377cf6c9c4db4a8c5ea31a7d400b081095be7100: Status 404 returned error can't find the container with id b33e305f2b8b2e902c766c6b377cf6c9c4db4a8c5ea31a7d400b081095be7100
Oct 29 22:46:29.282910 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:29.282877    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:29.411452 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:29.411365    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:29.685892 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:29.685822    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr" event={"ID":"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f","Type":"ContainerStarted","Data":"b33e305f2b8b2e902c766c6b377cf6c9c4db4a8c5ea31a7d400b081095be7100"}
Oct 29 22:46:29.737540 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:29.737468    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:29.867744 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:29.867685    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:31.659594 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:31.659563    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:31.699896 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:31.699856    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx" event={"ID":"21d3b0f9-5747-40c0-b610-0efc8e08a6ce","Type":"ContainerStarted","Data":"8f0630e401163233dd797b8a900e5caa9a988d0512e57a9f928aa71f967a0485"}
Oct 29 22:46:31.700077 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:31.699972    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx"
Oct 29 22:46:31.742153 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:31.742100    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx" podStartSLOduration=1.639224134 podStartE2EDuration="5.742086444s" totalImagesPullingTime="4.10286231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:31.741750716 +0000 UTC m=+195.084699462" watchObservedRunningTime="2026-10-29 22:46:31.742086444 +0000 UTC m=+195.085035189"
Oct 29 22:46:32.080208 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:32.080168    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:32.468233 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:32.468201    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:32.705668 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:32.705636    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr" event={"ID":"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f","Type":"ContainerStarted","Data":"59aea24939b85e65d8c734ecbfa0689a9b13473098c5dafc8d1ed21e74733b45"}
Oct 29 22:46:32.720289 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:32.720220    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bhpvw"]
Oct 29 22:46:32.727042 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:32.727016    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bhpvw"
Oct 29 22:46:32.749398 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:32.749373    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bhpvw"]
Oct 29 22:46:32.750528 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:32.750510    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:46:32.750591 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:32.750510    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-zksz5\""
Oct 29 22:46:32.750591 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:32.750558    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 29 22:46:32.806451 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:32.806415    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-758wq\" (UniqueName: \"kubernetes.io/projected/d84a20ab-bc6a-4be6-8dd4-aa972e3dad1e-kube-api-access-758wq\") pod \"jobset-operator-674cd59dc5-bhpvw\" (UID: \"d84a20ab-bc6a-4be6-8dd4-aa972e3dad1e\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bhpvw"
Oct 29 22:46:32.806631 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:32.806481    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d84a20ab-bc6a-4be6-8dd4-aa972e3dad1e-tmp\") pod \"jobset-operator-674cd59dc5-bhpvw\" (UID: \"d84a20ab-bc6a-4be6-8dd4-aa972e3dad1e\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bhpvw"
Oct 29 22:46:32.907617 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:32.907579    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-758wq\" (UniqueName: \"kubernetes.io/projected/d84a20ab-bc6a-4be6-8dd4-aa972e3dad1e-kube-api-access-758wq\") pod \"jobset-operator-674cd59dc5-bhpvw\" (UID: \"d84a20ab-bc6a-4be6-8dd4-aa972e3dad1e\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bhpvw"
Oct 29 22:46:32.907764 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:32.907629    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d84a20ab-bc6a-4be6-8dd4-aa972e3dad1e-tmp\") pod \"jobset-operator-674cd59dc5-bhpvw\" (UID: \"d84a20ab-bc6a-4be6-8dd4-aa972e3dad1e\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bhpvw"
Oct 29 22:46:32.908080 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:32.908061    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d84a20ab-bc6a-4be6-8dd4-aa972e3dad1e-tmp\") pod \"jobset-operator-674cd59dc5-bhpvw\" (UID: \"d84a20ab-bc6a-4be6-8dd4-aa972e3dad1e\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bhpvw"
Oct 29 22:46:32.934271 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:32.934243    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-758wq\" (UniqueName: \"kubernetes.io/projected/d84a20ab-bc6a-4be6-8dd4-aa972e3dad1e-kube-api-access-758wq\") pod \"jobset-operator-674cd59dc5-bhpvw\" (UID: \"d84a20ab-bc6a-4be6-8dd4-aa972e3dad1e\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bhpvw"
Oct 29 22:46:33.045290 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:33.045222    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bhpvw"
Oct 29 22:46:33.347715 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:33.347615    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bhpvw"]
Oct 29 22:46:33.355427 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:46:33.355399    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd84a20ab_bc6a_4be6_8dd4_aa972e3dad1e.slice/crio-ada2dc47c20b373e03b2c5b9399ca614d7c68f80365b80e87a21f150f70718b0 WatchSource:0}: Error finding container ada2dc47c20b373e03b2c5b9399ca614d7c68f80365b80e87a21f150f70718b0: Status 404 returned error can't find the container with id ada2dc47c20b373e03b2c5b9399ca614d7c68f80365b80e87a21f150f70718b0
Oct 29 22:46:33.710950 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:33.710910    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bhpvw" event={"ID":"d84a20ab-bc6a-4be6-8dd4-aa972e3dad1e","Type":"ContainerStarted","Data":"ada2dc47c20b373e03b2c5b9399ca614d7c68f80365b80e87a21f150f70718b0"}
Oct 29 22:46:33.712341 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:33.712317    5428 generic.go:356] "Generic (PLEG): container finished" podID="050c87b1-f0de-47c6-82cd-2c2eac0e0c1f" containerID="59aea24939b85e65d8c734ecbfa0689a9b13473098c5dafc8d1ed21e74733b45" exitCode=0
Oct 29 22:46:33.712468 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:33.712374    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr" event={"ID":"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f","Type":"ContainerDied","Data":"59aea24939b85e65d8c734ecbfa0689a9b13473098c5dafc8d1ed21e74733b45"}
Oct 29 22:46:34.718893 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:34.718856    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr" event={"ID":"050c87b1-f0de-47c6-82cd-2c2eac0e0c1f","Type":"ContainerStarted","Data":"f9c3e4f2787c49c4d22dddb9983d12aaf81ae916cec5aec8e859a207fe91301f"}
Oct 29 22:46:34.719338 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:34.718955    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:34.817888 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:34.817819    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr" podStartSLOduration=3.053554013 podStartE2EDuration="6.81778876s" totalImagesPullingTime="3.268602081s" totalInitContainerRuntime="495.632666ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:34.807423885 +0000 UTC m=+198.150372667" watchObservedRunningTime="2026-10-29 22:46:34.81778876 +0000 UTC m=+198.160737506"
Oct 29 22:46:34.903810 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:34.903771    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:35.054864 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:35.054768    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:35.226782 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:35.226754    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:35.724170 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:35.724131    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bhpvw" event={"ID":"d84a20ab-bc6a-4be6-8dd4-aa972e3dad1e","Type":"ContainerStarted","Data":"9c7ec782de8951e3d427fd8bbdca3b713dfc2453019737cce40f157eef212854"}
Oct 29 22:46:35.766491 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:35.766441    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bhpvw" podStartSLOduration=1.9055961300000002 podStartE2EDuration="3.766427501s" totalImagesPullingTime="1.860831371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:35.764772983 +0000 UTC m=+199.107721727" watchObservedRunningTime="2026-10-29 22:46:35.766427501 +0000 UTC m=+199.109376246"
Oct 29 22:46:35.810221 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:35.810191    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:35.934737 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:35.934703    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:36.082340 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:36.082267    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-dqxbf"]
Oct 29 22:46:36.085393 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:36.085374    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dqxbf"
Oct 29 22:46:36.098583 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:36.098565    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ltcbq\""
Oct 29 22:46:36.113468 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:36.113446    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-dqxbf"]
Oct 29 22:46:36.134332 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:36.134299    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwnq7\" (UniqueName: \"kubernetes.io/projected/7f6616af-b4f8-4cd9-9cd1-c356b28d97c4-kube-api-access-wwnq7\") pod \"authorino-operator-7b7bb4bc77-dqxbf\" (UID: \"7f6616af-b4f8-4cd9-9cd1-c356b28d97c4\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-dqxbf"
Oct 29 22:46:36.235461 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:36.235426    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwnq7\" (UniqueName: \"kubernetes.io/projected/7f6616af-b4f8-4cd9-9cd1-c356b28d97c4-kube-api-access-wwnq7\") pod \"authorino-operator-7b7bb4bc77-dqxbf\" (UID: \"7f6616af-b4f8-4cd9-9cd1-c356b28d97c4\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-dqxbf"
Oct 29 22:46:36.257006 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:36.256975    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwnq7\" (UniqueName: \"kubernetes.io/projected/7f6616af-b4f8-4cd9-9cd1-c356b28d97c4-kube-api-access-wwnq7\") pod \"authorino-operator-7b7bb4bc77-dqxbf\" (UID: \"7f6616af-b4f8-4cd9-9cd1-c356b28d97c4\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-dqxbf"
Oct 29 22:46:36.397676 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:36.397652    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dqxbf"
Oct 29 22:46:36.550818 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:36.550777    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:36.696163 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:36.696078    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-dqxbf"]
Oct 29 22:46:36.700150 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:46:36.700124    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6616af_b4f8_4cd9_9cd1_c356b28d97c4.slice/crio-938ac9443e3404d7b55f9bb26b0bf80fb7e3b1fecb0550d4861b45dc2f8f2390 WatchSource:0}: Error finding container 938ac9443e3404d7b55f9bb26b0bf80fb7e3b1fecb0550d4861b45dc2f8f2390: Status 404 returned error can't find the container with id 938ac9443e3404d7b55f9bb26b0bf80fb7e3b1fecb0550d4861b45dc2f8f2390
Oct 29 22:46:36.729102 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:36.729080    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dqxbf" event={"ID":"7f6616af-b4f8-4cd9-9cd1-c356b28d97c4","Type":"ContainerStarted","Data":"938ac9443e3404d7b55f9bb26b0bf80fb7e3b1fecb0550d4861b45dc2f8f2390"}
Oct 29 22:46:37.052086 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:37.051995    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:37.493006 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:37.492965    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:38.740375 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:38.740338    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dqxbf" event={"ID":"7f6616af-b4f8-4cd9-9cd1-c356b28d97c4","Type":"ContainerStarted","Data":"a16835bae9a702674a555708f8ecbf0cb3a4da5703a5d11eca8a0687d70705ac"}
Oct 29 22:46:38.740746 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:38.740520    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dqxbf"
Oct 29 22:46:38.788564 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:38.788519    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dqxbf" podStartSLOduration=1.784521712 podStartE2EDuration="2.788507353s" totalImagesPullingTime="1.003985641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:38.777768835 +0000 UTC m=+202.120717580" watchObservedRunningTime="2026-10-29 22:46:38.788507353 +0000 UTC m=+202.131456097"
Oct 29 22:46:40.731966 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:40.731931    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:42.438563 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:42.438527    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:42.708314 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:42.708232    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx"
Oct 29 22:46:43.750625 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:43.750585    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:43.861301 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:43.861265    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:44.111730 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:44.111648    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:45.726843 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:45.726815    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-844ffc67f7-882vr"
Oct 29 22:46:46.768497 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:46.768464    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:47.097353 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:47.097278    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x"]
Oct 29 22:46:47.099665 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:47.099649    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x"
Oct 29 22:46:47.115670 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:47.115647    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:46:47.115670 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:47.115650    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 29 22:46:47.126374 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:47.126355    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-5gjmm\""
Oct 29 22:46:47.155608 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:47.155577    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x"]
Oct 29 22:46:47.229863 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:47.229832    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gkq2\" (UniqueName: \"kubernetes.io/projected/46bc949c-5680-4bbf-8842-abc5d8d4d4f7-kube-api-access-8gkq2\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x\" (UID: \"46bc949c-5680-4bbf-8842-abc5d8d4d4f7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x"
Oct 29 22:46:47.229993 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:47.229902    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46bc949c-5680-4bbf-8842-abc5d8d4d4f7-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x\" (UID: \"46bc949c-5680-4bbf-8842-abc5d8d4d4f7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x"
Oct 29 22:46:47.237103 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:47.237079    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:47.330678 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:47.330649    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46bc949c-5680-4bbf-8842-abc5d8d4d4f7-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x\" (UID: \"46bc949c-5680-4bbf-8842-abc5d8d4d4f7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x"
Oct 29 22:46:47.330856 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:47.330691    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gkq2\" (UniqueName: \"kubernetes.io/projected/46bc949c-5680-4bbf-8842-abc5d8d4d4f7-kube-api-access-8gkq2\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x\" (UID: \"46bc949c-5680-4bbf-8842-abc5d8d4d4f7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x"
Oct 29 22:46:47.331073 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:47.331055    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46bc949c-5680-4bbf-8842-abc5d8d4d4f7-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x\" (UID: \"46bc949c-5680-4bbf-8842-abc5d8d4d4f7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x"
Oct 29 22:46:47.356317 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:47.356257    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gkq2\" (UniqueName: \"kubernetes.io/projected/46bc949c-5680-4bbf-8842-abc5d8d4d4f7-kube-api-access-8gkq2\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x\" (UID: \"46bc949c-5680-4bbf-8842-abc5d8d4d4f7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x"
Oct 29 22:46:47.366496 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:46:47.366469    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:46:47.410879 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:47.410851    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x"
Oct 29 22:46:47.436463 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:47.436437    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:47.741681 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:47.741643    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x"]
Oct 29 22:46:47.745899 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:46:47.745874    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bc949c_5680_4bbf_8842_abc5d8d4d4f7.slice/crio-375a7b976c6b67518dfcdf3491d14f5f5f289a51484aa973be1a0b66b8cac6eb WatchSource:0}: Error finding container 375a7b976c6b67518dfcdf3491d14f5f5f289a51484aa973be1a0b66b8cac6eb: Status 404 returned error can't find the container with id 375a7b976c6b67518dfcdf3491d14f5f5f289a51484aa973be1a0b66b8cac6eb
Oct 29 22:46:47.781147 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:47.781116    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x" event={"ID":"46bc949c-5680-4bbf-8842-abc5d8d4d4f7","Type":"ContainerStarted","Data":"375a7b976c6b67518dfcdf3491d14f5f5f289a51484aa973be1a0b66b8cac6eb"}
Oct 29 22:46:47.871918 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:47.871886    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:48.016208 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:48.016112    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:48.747712 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:48.747676    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:48.960454 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:48.960421    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:49.747545 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:49.747518    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dqxbf"
Oct 29 22:46:56.059024 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:56.058975    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:56.178859 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:56.178824    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:57.912179 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:46:57.912136    5428 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image: copying system image from manifest list: reading signatures: Get \"https://access.redhat.com/webassets/docker/content/sigstore/cert-manager/cert-manager-operator-rhel9@sha256=403674a15e95ce09e0823b49c961f09a070e36c33d051c8a97e9cafc260da397/signature-1\": net/http: TLS handshake timeout" image="registry.redhat.io/cert-manager/cert-manager-operator-rhel9@sha256:207804f9237b0c2875f5252997312230c88726b5271c6419320958078dd7b7fd"
Oct 29 22:46:57.912564 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:46:57.912355    5428 kuberuntime_manager.go:1707] "Unhandled Error" err="container cert-manager-operator start failed in pod cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x_cert-manager-operator(46bc949c-5680-4bbf-8842-abc5d8d4d4f7): ErrImagePull: unable to pull image: copying system image from manifest list: reading signatures: Get \"https://access.redhat.com/webassets/docker/content/sigstore/cert-manager/cert-manager-operator-rhel9@sha256=403674a15e95ce09e0823b49c961f09a070e36c33d051c8a97e9cafc260da397/signature-1\": net/http: TLS handshake timeout" logger="UnhandledError"
Oct 29 22:46:57.912564 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:46:57.912393    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-operator\" with ErrImagePull: \"unable to pull image: copying system image from manifest list: reading signatures: Get \\\"https://access.redhat.com/webassets/docker/content/sigstore/cert-manager/cert-manager-operator-rhel9@sha256=403674a15e95ce09e0823b49c961f09a070e36c33d051c8a97e9cafc260da397/signature-1\\\": net/http: TLS handshake timeout\"" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x" podUID="46bc949c-5680-4bbf-8842-abc5d8d4d4f7"
Oct 29 22:46:58.832050 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:46:58.832019    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-operator\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/cert-manager/cert-manager-operator-rhel9@sha256:207804f9237b0c2875f5252997312230c88726b5271c6419320958078dd7b7fd\\\": ErrImagePull: unable to pull image: copying system image from manifest list: reading signatures: Get \\\"https://access.redhat.com/webassets/docker/content/sigstore/cert-manager/cert-manager-operator-rhel9@sha256=403674a15e95ce09e0823b49c961f09a070e36c33d051c8a97e9cafc260da397/signature-1\\\": net/http: TLS handshake timeout\"" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x" podUID="46bc949c-5680-4bbf-8842-abc5d8d4d4f7"
Oct 29 22:46:59.804583 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:59.804552    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx"]
Oct 29 22:46:59.805051 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:46:59.804751    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx" podUID="21d3b0f9-5747-40c0-b610-0efc8e08a6ce" containerName="manager" containerID="cri-o://8f0630e401163233dd797b8a900e5caa9a988d0512e57a9f928aa71f967a0485" gracePeriod=10
Oct 29 22:47:00.039538 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:00.039513    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx"
Oct 29 22:47:00.137606 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:00.137574    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d3b0f9-5747-40c0-b610-0efc8e08a6ce-kube-api-access-jfqc4\" (UniqueName: \"kubernetes.io/projected/21d3b0f9-5747-40c0-b610-0efc8e08a6ce-kube-api-access-jfqc4\") pod \"21d3b0f9-5747-40c0-b610-0efc8e08a6ce\" (UID: \"21d3b0f9-5747-40c0-b610-0efc8e08a6ce\") "
Oct 29 22:47:00.137787 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:00.137639    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d3b0f9-5747-40c0-b610-0efc8e08a6ce-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/21d3b0f9-5747-40c0-b610-0efc8e08a6ce-extensions-socket-volume\") pod \"21d3b0f9-5747-40c0-b610-0efc8e08a6ce\" (UID: \"21d3b0f9-5747-40c0-b610-0efc8e08a6ce\") "
Oct 29 22:47:00.138143 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:00.138120    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d3b0f9-5747-40c0-b610-0efc8e08a6ce-extensions-socket-volume" pod "21d3b0f9-5747-40c0-b610-0efc8e08a6ce" (UID: "21d3b0f9-5747-40c0-b610-0efc8e08a6ce"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:00.139706 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:00.139683    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d3b0f9-5747-40c0-b610-0efc8e08a6ce-kube-api-access-jfqc4" pod "21d3b0f9-5747-40c0-b610-0efc8e08a6ce" (UID: "21d3b0f9-5747-40c0-b610-0efc8e08a6ce"). InnerVolumeSpecName "kube-api-access-jfqc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:00.238960 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:00.238930    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfqc4\" (UniqueName: \"kubernetes.io/projected/21d3b0f9-5747-40c0-b610-0efc8e08a6ce-kube-api-access-jfqc4\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:47:00.238960 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:00.238954    5428 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/21d3b0f9-5747-40c0-b610-0efc8e08a6ce-extensions-socket-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:47:00.840738 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:00.840707    5428 generic.go:356] "Generic (PLEG): container finished" podID="21d3b0f9-5747-40c0-b610-0efc8e08a6ce" containerID="8f0630e401163233dd797b8a900e5caa9a988d0512e57a9f928aa71f967a0485" exitCode=0
Oct 29 22:47:00.841141 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:00.840754    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx"
Oct 29 22:47:00.841141 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:00.840781    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx" event={"ID":"21d3b0f9-5747-40c0-b610-0efc8e08a6ce","Type":"ContainerDied","Data":"8f0630e401163233dd797b8a900e5caa9a988d0512e57a9f928aa71f967a0485"}
Oct 29 22:47:00.841141 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:00.840846    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx" event={"ID":"21d3b0f9-5747-40c0-b610-0efc8e08a6ce","Type":"ContainerDied","Data":"2782a5db87fdf01a03b1235ce0734cfef88864a7922fd2c816223c24033c327b"}
Oct 29 22:47:00.841141 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:00.840873    5428 scope.go:122] "RemoveContainer" containerID="8f0630e401163233dd797b8a900e5caa9a988d0512e57a9f928aa71f967a0485"
Oct 29 22:47:00.849359 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:00.849335    5428 scope.go:122] "RemoveContainer" containerID="8f0630e401163233dd797b8a900e5caa9a988d0512e57a9f928aa71f967a0485"
Oct 29 22:47:00.849635 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:47:00.849608    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0630e401163233dd797b8a900e5caa9a988d0512e57a9f928aa71f967a0485\": container with ID starting with 8f0630e401163233dd797b8a900e5caa9a988d0512e57a9f928aa71f967a0485 not found: ID does not exist" containerID="8f0630e401163233dd797b8a900e5caa9a988d0512e57a9f928aa71f967a0485"
Oct 29 22:47:00.849740 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:00.849641    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0630e401163233dd797b8a900e5caa9a988d0512e57a9f928aa71f967a0485"} err="failed to get container status \"8f0630e401163233dd797b8a900e5caa9a988d0512e57a9f928aa71f967a0485\": rpc error: code = NotFound desc = could not find container \"8f0630e401163233dd797b8a900e5caa9a988d0512e57a9f928aa71f967a0485\": container with ID starting with 8f0630e401163233dd797b8a900e5caa9a988d0512e57a9f928aa71f967a0485 not found: ID does not exist"
Oct 29 22:47:00.903719 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:00.903691    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx"]
Oct 29 22:47:00.913791 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:00.913767    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-wtqwx"]
Oct 29 22:47:01.360832 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:01.360782    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:01.491649 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:01.491618    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:02.340142 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:02.340109    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:02.475834 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:02.475785    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:02.708421 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:02.708385    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:02.774264 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:02.774234    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21d3b0f9-5747-40c0-b610-0efc8e08a6ce" path="/var/lib/kubelet/pods/21d3b0f9-5747-40c0-b610-0efc8e08a6ce/volumes"
Oct 29 22:47:02.944067 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:02.944037    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-67f887f896-mv4wq"]
Oct 29 22:47:02.944346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:02.944334    5428 cpu_manager.go:414] "containerMap: removing container" podUID="21d3b0f9-5747-40c0-b610-0efc8e08a6ce" containerName="manager"
Oct 29 22:47:02.944401 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:02.944347    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d3b0f9-5747-40c0-b610-0efc8e08a6ce" containerName="manager"
Oct 29 22:47:02.944401 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:02.944397    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d3b0f9-5747-40c0-b610-0efc8e08a6ce" containerName="manager"
Oct 29 22:47:02.950964 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:02.950935    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:03.000913 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.000846    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67f887f896-mv4wq"]
Oct 29 22:47:03.001042 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.000941    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67f887f896-mv4wq"
Oct 29 22:47:03.086061 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.086029    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6mfk4\""
Oct 29 22:47:03.086257 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.086241    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 29 22:47:03.086314 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.086242    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 29 22:47:03.155559 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.155529    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:03.165707 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.165679    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fafd9f2b-e714-4a9f-9ac9-20031efbf034-tls-certs\") pod \"kube-auth-proxy-67f887f896-mv4wq\" (UID: \"fafd9f2b-e714-4a9f-9ac9-20031efbf034\") " pod="openshift-ingress/kube-auth-proxy-67f887f896-mv4wq"
Oct 29 22:47:03.165833 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.165716    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fafd9f2b-e714-4a9f-9ac9-20031efbf034-tmp\") pod \"kube-auth-proxy-67f887f896-mv4wq\" (UID: \"fafd9f2b-e714-4a9f-9ac9-20031efbf034\") " pod="openshift-ingress/kube-auth-proxy-67f887f896-mv4wq"
Oct 29 22:47:03.165833 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.165755    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2k72\" (UniqueName: \"kubernetes.io/projected/fafd9f2b-e714-4a9f-9ac9-20031efbf034-kube-api-access-p2k72\") pod \"kube-auth-proxy-67f887f896-mv4wq\" (UID: \"fafd9f2b-e714-4a9f-9ac9-20031efbf034\") " pod="openshift-ingress/kube-auth-proxy-67f887f896-mv4wq"
Oct 29 22:47:03.199697 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.199672    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"]
Oct 29 22:47:03.216349 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.216330    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.228659 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.228635    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"]
Oct 29 22:47:03.232152 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.232126    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-hb9zm\""
Oct 29 22:47:03.266854 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.266766    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fafd9f2b-e714-4a9f-9ac9-20031efbf034-tls-certs\") pod \"kube-auth-proxy-67f887f896-mv4wq\" (UID: \"fafd9f2b-e714-4a9f-9ac9-20031efbf034\") " pod="openshift-ingress/kube-auth-proxy-67f887f896-mv4wq"
Oct 29 22:47:03.266854 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.266835    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fafd9f2b-e714-4a9f-9ac9-20031efbf034-tmp\") pod \"kube-auth-proxy-67f887f896-mv4wq\" (UID: \"fafd9f2b-e714-4a9f-9ac9-20031efbf034\") " pod="openshift-ingress/kube-auth-proxy-67f887f896-mv4wq"
Oct 29 22:47:03.267031 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.266888    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2k72\" (UniqueName: \"kubernetes.io/projected/fafd9f2b-e714-4a9f-9ac9-20031efbf034-kube-api-access-p2k72\") pod \"kube-auth-proxy-67f887f896-mv4wq\" (UID: \"fafd9f2b-e714-4a9f-9ac9-20031efbf034\") " pod="openshift-ingress/kube-auth-proxy-67f887f896-mv4wq"
Oct 29 22:47:03.268996 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.268966    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fafd9f2b-e714-4a9f-9ac9-20031efbf034-tmp\") pod \"kube-auth-proxy-67f887f896-mv4wq\" (UID: \"fafd9f2b-e714-4a9f-9ac9-20031efbf034\") " pod="openshift-ingress/kube-auth-proxy-67f887f896-mv4wq"
Oct 29 22:47:03.269179 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.269161    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fafd9f2b-e714-4a9f-9ac9-20031efbf034-tls-certs\") pod \"kube-auth-proxy-67f887f896-mv4wq\" (UID: \"fafd9f2b-e714-4a9f-9ac9-20031efbf034\") " pod="openshift-ingress/kube-auth-proxy-67f887f896-mv4wq"
Oct 29 22:47:03.277497 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.277468    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:03.288044 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.288027    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2k72\" (UniqueName: \"kubernetes.io/projected/fafd9f2b-e714-4a9f-9ac9-20031efbf034-kube-api-access-p2k72\") pod \"kube-auth-proxy-67f887f896-mv4wq\" (UID: \"fafd9f2b-e714-4a9f-9ac9-20031efbf034\") " pod="openshift-ingress/kube-auth-proxy-67f887f896-mv4wq"
Oct 29 22:47:03.328016 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.327986    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67f887f896-mv4wq"
Oct 29 22:47:03.367433 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.367407    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0b257f-c278-4918-a13a-284e1fd77231-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.367831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.367445    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsmph\" (UniqueName: \"kubernetes.io/projected/6f0b257f-c278-4918-a13a-284e1fd77231-kube-api-access-nsmph\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.367831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.367550    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0b257f-c278-4918-a13a-284e1fd77231-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.367831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.367588    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0b257f-c278-4918-a13a-284e1fd77231-istio-token\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.367831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.367627    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0b257f-c278-4918-a13a-284e1fd77231-istio-data\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.367831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.367692    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0b257f-c278-4918-a13a-284e1fd77231-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.367831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.367717    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0b257f-c278-4918-a13a-284e1fd77231-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.367831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.367742    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0b257f-c278-4918-a13a-284e1fd77231-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.367831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.367763    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0b257f-c278-4918-a13a-284e1fd77231-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.429045 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.429011    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:03.468241 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.468212    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0b257f-c278-4918-a13a-284e1fd77231-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.468401 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.468250    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0b257f-c278-4918-a13a-284e1fd77231-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.468401 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.468277    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsmph\" (UniqueName: \"kubernetes.io/projected/6f0b257f-c278-4918-a13a-284e1fd77231-kube-api-access-nsmph\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.468401 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.468327    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0b257f-c278-4918-a13a-284e1fd77231-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.468401 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.468367    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0b257f-c278-4918-a13a-284e1fd77231-istio-token\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.468633 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.468418    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0b257f-c278-4918-a13a-284e1fd77231-istio-data\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.468633 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.468457    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0b257f-c278-4918-a13a-284e1fd77231-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.468633 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.468484    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0b257f-c278-4918-a13a-284e1fd77231-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.468633 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.468517    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0b257f-c278-4918-a13a-284e1fd77231-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.468821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.468635    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0b257f-c278-4918-a13a-284e1fd77231-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.468821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.468679    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0b257f-c278-4918-a13a-284e1fd77231-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.469028 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.468983    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0b257f-c278-4918-a13a-284e1fd77231-istio-data\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.469028 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.469000    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0b257f-c278-4918-a13a-284e1fd77231-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.469206 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.469183    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0b257f-c278-4918-a13a-284e1fd77231-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.470721 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.470700    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0b257f-c278-4918-a13a-284e1fd77231-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.471571 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.471553    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0b257f-c278-4918-a13a-284e1fd77231-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.489775 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.489748    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsmph\" (UniqueName: \"kubernetes.io/projected/6f0b257f-c278-4918-a13a-284e1fd77231-kube-api-access-nsmph\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.489875 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.489858    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0b257f-c278-4918-a13a-284e1fd77231-istio-token\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7\" (UID: \"6f0b257f-c278-4918-a13a-284e1fd77231\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.547050 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.546973    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:03.681348 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.681314    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67f887f896-mv4wq"]
Oct 29 22:47:03.684736 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:47:03.684710    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafd9f2b_e714_4a9f_9ac9_20031efbf034.slice/crio-0365471096e740005e21fb227b3aaf453eba7661248455ec1499a1c3f032f643 WatchSource:0}: Error finding container 0365471096e740005e21fb227b3aaf453eba7661248455ec1499a1c3f032f643: Status 404 returned error can't find the container with id 0365471096e740005e21fb227b3aaf453eba7661248455ec1499a1c3f032f643
Oct 29 22:47:03.855588 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.855513    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67f887f896-mv4wq" event={"ID":"fafd9f2b-e714-4a9f-9ac9-20031efbf034","Type":"ContainerStarted","Data":"0365471096e740005e21fb227b3aaf453eba7661248455ec1499a1c3f032f643"}
Oct 29 22:47:03.907071 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.907039    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"]
Oct 29 22:47:03.910922 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:47:03.910896    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0b257f_c278_4918_a13a_284e1fd77231.slice/crio-35467a3206aa4a65dafb1305425e85c7bda3349f8141e10b5261ab8769a1a20d WatchSource:0}: Error finding container 35467a3206aa4a65dafb1305425e85c7bda3349f8141e10b5261ab8769a1a20d: Status 404 returned error can't find the container with id 35467a3206aa4a65dafb1305425e85c7bda3349f8141e10b5261ab8769a1a20d
Oct 29 22:47:03.912989 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.912958    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:47:03.913054 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.913037    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:47:03.913092 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:03.913065    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:47:04.290129 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:04.290082    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:04.412589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:04.412553    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:04.716080 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:04.716048    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:04.862544 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:04.862504    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7" event={"ID":"6f0b257f-c278-4918-a13a-284e1fd77231","Type":"ContainerStarted","Data":"ede912bc171f038ec73d5a2690cb1758dd744ae5b378e9ea0c829ab3527dbe81"}
Oct 29 22:47:04.862737 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:04.862569    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7" event={"ID":"6f0b257f-c278-4918-a13a-284e1fd77231","Type":"ContainerStarted","Data":"35467a3206aa4a65dafb1305425e85c7bda3349f8141e10b5261ab8769a1a20d"}
Oct 29 22:47:04.904442 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:04.904212    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7" podStartSLOduration=1.9042002820000001 podStartE2EDuration="1.904200282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:04.90312486 +0000 UTC m=+228.246073643" watchObservedRunningTime="2026-10-29 22:47:04.904200282 +0000 UTC m=+228.247149028"
Oct 29 22:47:05.548273 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:05.548230    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:05.629337 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:05.554760    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:05.629337 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:05.569241    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:05.868092 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:05.868064    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:05.869155 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:05.869132    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7"
Oct 29 22:47:06.874152 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:06.874121    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67f887f896-mv4wq" event={"ID":"fafd9f2b-e714-4a9f-9ac9-20031efbf034","Type":"ContainerStarted","Data":"76f43df76fe6188666d2ab2c5e1af0b6960e49fcae1733ff2e97fdc288188386"}
Oct 29 22:47:06.914055 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:06.913998    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-67f887f896-mv4wq" podStartSLOduration=2.447773207 podStartE2EDuration="4.913983801s" totalImagesPullingTime="2.466210594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:06.913037774 +0000 UTC m=+230.255986518" watchObservedRunningTime="2026-10-29 22:47:06.913983801 +0000 UTC m=+230.256932598"
Oct 29 22:47:09.171552 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:09.171519    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:09.290815 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:09.290779    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:10.839686 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:10.839651    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"]
Oct 29 22:47:10.842403 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:10.842376    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"
Oct 29 22:47:10.858146 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:10.858126    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 29 22:47:10.869667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:10.869644    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-qvb26\""
Oct 29 22:47:10.869667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:10.869645    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 29 22:47:10.891140 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:10.891101    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 29 22:47:10.892756 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:10.892726    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"]
Oct 29 22:47:11.034653 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:11.034614    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d668198-fafe-4276-9474-e55b192582bf-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-6pqtr\" (UID: \"6d668198-fafe-4276-9474-e55b192582bf\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"
Oct 29 22:47:11.034864 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:11.034739    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6d668198-fafe-4276-9474-e55b192582bf-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-6pqtr\" (UID: \"6d668198-fafe-4276-9474-e55b192582bf\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"
Oct 29 22:47:11.034864 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:11.034847    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcfnl\" (UniqueName: \"kubernetes.io/projected/6d668198-fafe-4276-9474-e55b192582bf-kube-api-access-pcfnl\") pod \"kserve-module-controller-manager-7566cf5fc8-6pqtr\" (UID: \"6d668198-fafe-4276-9474-e55b192582bf\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"
Oct 29 22:47:11.034993 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:11.034915    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d668198-fafe-4276-9474-e55b192582bf-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-6pqtr\" (UID: \"6d668198-fafe-4276-9474-e55b192582bf\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"
Oct 29 22:47:11.136179 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:11.136154    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d668198-fafe-4276-9474-e55b192582bf-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-6pqtr\" (UID: \"6d668198-fafe-4276-9474-e55b192582bf\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"
Oct 29 22:47:11.136298 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:11.136209    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d668198-fafe-4276-9474-e55b192582bf-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-6pqtr\" (UID: \"6d668198-fafe-4276-9474-e55b192582bf\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"
Oct 29 22:47:11.136298 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:11.136268    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6d668198-fafe-4276-9474-e55b192582bf-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-6pqtr\" (UID: \"6d668198-fafe-4276-9474-e55b192582bf\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"
Oct 29 22:47:11.136398 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:11.136297    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcfnl\" (UniqueName: \"kubernetes.io/projected/6d668198-fafe-4276-9474-e55b192582bf-kube-api-access-pcfnl\") pod \"kserve-module-controller-manager-7566cf5fc8-6pqtr\" (UID: \"6d668198-fafe-4276-9474-e55b192582bf\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"
Oct 29 22:47:11.136575 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:11.136549    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d668198-fafe-4276-9474-e55b192582bf-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-6pqtr\" (UID: \"6d668198-fafe-4276-9474-e55b192582bf\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"
Oct 29 22:47:11.136612 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:11.136564    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6d668198-fafe-4276-9474-e55b192582bf-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-6pqtr\" (UID: \"6d668198-fafe-4276-9474-e55b192582bf\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"
Oct 29 22:47:11.137037 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:11.137019    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d668198-fafe-4276-9474-e55b192582bf-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-6pqtr\" (UID: \"6d668198-fafe-4276-9474-e55b192582bf\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"
Oct 29 22:47:11.160978 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:11.160957    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcfnl\" (UniqueName: \"kubernetes.io/projected/6d668198-fafe-4276-9474-e55b192582bf-kube-api-access-pcfnl\") pod \"kserve-module-controller-manager-7566cf5fc8-6pqtr\" (UID: \"6d668198-fafe-4276-9474-e55b192582bf\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"
Oct 29 22:47:11.455279 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:11.455246    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"
Oct 29 22:47:11.772983 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:11.772955    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"]
Oct 29 22:47:11.774499 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:47:11.774472    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d668198_fafe_4276_9474_e55b192582bf.slice/crio-891c33f12e728c12a1d07cd5be767e1ebff3bb18daa7ed68fdf11f34ab922329 WatchSource:0}: Error finding container 891c33f12e728c12a1d07cd5be767e1ebff3bb18daa7ed68fdf11f34ab922329: Status 404 returned error can't find the container with id 891c33f12e728c12a1d07cd5be767e1ebff3bb18daa7ed68fdf11f34ab922329
Oct 29 22:47:11.898490 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:11.898441    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x" event={"ID":"46bc949c-5680-4bbf-8842-abc5d8d4d4f7","Type":"ContainerStarted","Data":"f78c8aa95b0c552cc6b634e51ace9fe2d74de977f335801d8f072bc2a77b5402"}
Oct 29 22:47:11.899672 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:11.899644    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr" event={"ID":"6d668198-fafe-4276-9474-e55b192582bf","Type":"ContainerStarted","Data":"891c33f12e728c12a1d07cd5be767e1ebff3bb18daa7ed68fdf11f34ab922329"}
Oct 29 22:47:12.012319 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:12.012213    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bbq9x" podStartSLOduration=1.64571875 podStartE2EDuration="25.012199242s" totalImagesPullingTime="23.366480492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-10-29 22:47:11.965838152 +0000 UTC m=+235.308786896" watchObservedRunningTime="2026-10-29 22:47:12.012199242 +0000 UTC m=+235.355147986"
Oct 29 22:47:12.262605 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:12.262574    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:12.420711 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:12.420672    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:13.909513 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:13.909476    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr" event={"ID":"6d668198-fafe-4276-9474-e55b192582bf","Type":"ContainerStarted","Data":"910920ee973840cae325a7dc83d44a14082fb8c78cabf4863d59d86e0fa64242"}
Oct 29 22:47:13.909956 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:13.909572    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"
Oct 29 22:47:13.961511 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:13.961466    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr" podStartSLOduration=2.570761046 podStartE2EDuration="3.961454021s" totalImagesPullingTime="1.390692975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:13.960846698 +0000 UTC m=+237.303795444" watchObservedRunningTime="2026-10-29 22:47:13.961454021 +0000 UTC m=+237.304402766"
Oct 29 22:47:14.641826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:14.641780    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:14.872637 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:14.872604    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:15.697549 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:15.697512    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:15.819566 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:15.819537    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:15.990120 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:15.990030    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8r9f6"]
Oct 29 22:47:15.992171 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:15.992155    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8r9f6"
Oct 29 22:47:16.014732 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:16.014710    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fppr9\""
Oct 29 22:47:16.031195 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:16.031173    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 29 22:47:16.031195 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:16.031173    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 29 22:47:16.032620 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:16.032600    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8r9f6"]
Oct 29 22:47:16.082854 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:16.082828    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df0fa68c-cefe-4881-8d9e-e6c39ce46a3a-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8r9f6\" (UID: \"df0fa68c-cefe-4881-8d9e-e6c39ce46a3a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8r9f6"
Oct 29 22:47:16.082968 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:16.082865    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4hxx\" (UniqueName: \"kubernetes.io/projected/df0fa68c-cefe-4881-8d9e-e6c39ce46a3a-kube-api-access-z4hxx\") pod \"cert-manager-cainjector-5c57bbd4fc-8r9f6\" (UID: \"df0fa68c-cefe-4881-8d9e-e6c39ce46a3a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8r9f6"
Oct 29 22:47:16.184184 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:16.184153    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df0fa68c-cefe-4881-8d9e-e6c39ce46a3a-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8r9f6\" (UID: \"df0fa68c-cefe-4881-8d9e-e6c39ce46a3a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8r9f6"
Oct 29 22:47:16.184311 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:16.184189    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4hxx\" (UniqueName: \"kubernetes.io/projected/df0fa68c-cefe-4881-8d9e-e6c39ce46a3a-kube-api-access-z4hxx\") pod \"cert-manager-cainjector-5c57bbd4fc-8r9f6\" (UID: \"df0fa68c-cefe-4881-8d9e-e6c39ce46a3a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8r9f6"
Oct 29 22:47:16.205441 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:16.205411    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df0fa68c-cefe-4881-8d9e-e6c39ce46a3a-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8r9f6\" (UID: \"df0fa68c-cefe-4881-8d9e-e6c39ce46a3a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8r9f6"
Oct 29 22:47:16.205599 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:16.205580    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4hxx\" (UniqueName: \"kubernetes.io/projected/df0fa68c-cefe-4881-8d9e-e6c39ce46a3a-kube-api-access-z4hxx\") pod \"cert-manager-cainjector-5c57bbd4fc-8r9f6\" (UID: \"df0fa68c-cefe-4881-8d9e-e6c39ce46a3a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8r9f6"
Oct 29 22:47:16.302826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:16.302734    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8r9f6"
Oct 29 22:47:16.598250 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:16.598226    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8r9f6"]
Oct 29 22:47:16.600223 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:47:16.600187    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf0fa68c_cefe_4881_8d9e_e6c39ce46a3a.slice/crio-ac7d3bad66a006530ae5d0b17b18b67389e3fbd0102239221d0cb653f6d49735 WatchSource:0}: Error finding container ac7d3bad66a006530ae5d0b17b18b67389e3fbd0102239221d0cb653f6d49735: Status 404 returned error can't find the container with id ac7d3bad66a006530ae5d0b17b18b67389e3fbd0102239221d0cb653f6d49735
Oct 29 22:47:16.923534 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:16.923499    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8r9f6" event={"ID":"df0fa68c-cefe-4881-8d9e-e6c39ce46a3a","Type":"ContainerStarted","Data":"ac7d3bad66a006530ae5d0b17b18b67389e3fbd0102239221d0cb653f6d49735"}
Oct 29 22:47:17.374654 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:47:17.374610    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:47:17.400358 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:17.400320    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:17.657837 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:17.657739    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:17.778676 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:17.778616    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:18.345204 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:18.345170    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4dkx7"]
Oct 29 22:47:18.347606 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:18.347585    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4dkx7"
Oct 29 22:47:18.361014 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:18.360990    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xcpgt\""
Oct 29 22:47:18.373130 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:18.373061    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4dkx7"]
Oct 29 22:47:18.405987 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:18.405954    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsj9m\" (UniqueName: \"kubernetes.io/projected/66774a16-0c96-482f-b7ce-f9c0d23321b9-kube-api-access-hsj9m\") pod \"authorino-57f7f8659f-4dkx7\" (UID: \"66774a16-0c96-482f-b7ce-f9c0d23321b9\") " pod="kuadrant-system/authorino-57f7f8659f-4dkx7"
Oct 29 22:47:18.421896 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:18.421867    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:18.507021 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:18.506985    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsj9m\" (UniqueName: \"kubernetes.io/projected/66774a16-0c96-482f-b7ce-f9c0d23321b9-kube-api-access-hsj9m\") pod \"authorino-57f7f8659f-4dkx7\" (UID: \"66774a16-0c96-482f-b7ce-f9c0d23321b9\") " pod="kuadrant-system/authorino-57f7f8659f-4dkx7"
Oct 29 22:47:18.528792 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:18.528755    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsj9m\" (UniqueName: \"kubernetes.io/projected/66774a16-0c96-482f-b7ce-f9c0d23321b9-kube-api-access-hsj9m\") pod \"authorino-57f7f8659f-4dkx7\" (UID: \"66774a16-0c96-482f-b7ce-f9c0d23321b9\") " pod="kuadrant-system/authorino-57f7f8659f-4dkx7"
Oct 29 22:47:18.538139 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:18.538110    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:18.659871 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:18.659838    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4dkx7"
Oct 29 22:47:18.963945 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:18.963861    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4dkx7"]
Oct 29 22:47:18.967642 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:47:18.967612    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66774a16_0c96_482f_b7ce_f9c0d23321b9.slice/crio-21b445dcd23e24bb6d0676c46ddc3b4e719fe9e1a81193f7b149d31556a27d28 WatchSource:0}: Error finding container 21b445dcd23e24bb6d0676c46ddc3b4e719fe9e1a81193f7b149d31556a27d28: Status 404 returned error can't find the container with id 21b445dcd23e24bb6d0676c46ddc3b4e719fe9e1a81193f7b149d31556a27d28
Oct 29 22:47:19.940996 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:19.940952    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8r9f6" event={"ID":"df0fa68c-cefe-4881-8d9e-e6c39ce46a3a","Type":"ContainerStarted","Data":"41e169305df9a71b7921022b21ec35e095e0f5508344aa517eaecf0df4e1060f"}
Oct 29 22:47:19.942280 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:19.942249    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4dkx7" event={"ID":"66774a16-0c96-482f-b7ce-f9c0d23321b9","Type":"ContainerStarted","Data":"21b445dcd23e24bb6d0676c46ddc3b4e719fe9e1a81193f7b149d31556a27d28"}
Oct 29 22:47:19.998361 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:19.998292    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8r9f6" podStartSLOduration=2.47418224 podStartE2EDuration="4.998269887s" totalImagesPullingTime="2.524087647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:19.99565775 +0000 UTC m=+243.338606496" watchObservedRunningTime="2026-10-29 22:47:19.998269887 +0000 UTC m=+243.341218634"
Oct 29 22:47:20.279055 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:20.278966    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:20.396469 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:20.396429    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:20.948548 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:20.948465    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4dkx7" event={"ID":"66774a16-0c96-482f-b7ce-f9c0d23321b9","Type":"ContainerStarted","Data":"cb6be46c1f9063ae20e3bff8b8184c0388cd5595deecef014e92d76e6a1da053"}
Oct 29 22:47:21.000716 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:21.000666    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-4dkx7" podStartSLOduration=1.353260839 podStartE2EDuration="3.000652777s" totalImagesPullingTime="1.647391938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:20.999739159 +0000 UTC m=+244.342687907" watchObservedRunningTime="2026-10-29 22:47:21.000652777 +0000 UTC m=+244.343601523"
Oct 29 22:47:21.875438 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:21.875404    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:21.931189 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:21.931145    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4dkx7"]
Oct 29 22:47:22.249084 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:22.249001    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:22.358618 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:22.358585    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:22.565384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:22.565316    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-6zhb5"]
Oct 29 22:47:22.568518 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:22.568502    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-6zhb5"
Oct 29 22:47:22.594150 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:22.594125    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-m2s7d\""
Oct 29 22:47:22.596693 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:22.596668    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-6zhb5"]
Oct 29 22:47:22.643738 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:22.643711    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ca237c8-f54a-420c-a115-bf62a5708b41-bound-sa-token\") pod \"cert-manager-597b95cb84-6zhb5\" (UID: \"9ca237c8-f54a-420c-a115-bf62a5708b41\") " pod="cert-manager/cert-manager-597b95cb84-6zhb5"
Oct 29 22:47:22.643874 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:22.643750    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzmv9\" (UniqueName: \"kubernetes.io/projected/9ca237c8-f54a-420c-a115-bf62a5708b41-kube-api-access-lzmv9\") pod \"cert-manager-597b95cb84-6zhb5\" (UID: \"9ca237c8-f54a-420c-a115-bf62a5708b41\") " pod="cert-manager/cert-manager-597b95cb84-6zhb5"
Oct 29 22:47:22.744990 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:22.744955    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ca237c8-f54a-420c-a115-bf62a5708b41-bound-sa-token\") pod \"cert-manager-597b95cb84-6zhb5\" (UID: \"9ca237c8-f54a-420c-a115-bf62a5708b41\") " pod="cert-manager/cert-manager-597b95cb84-6zhb5"
Oct 29 22:47:22.745116 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:22.745005    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzmv9\" (UniqueName: \"kubernetes.io/projected/9ca237c8-f54a-420c-a115-bf62a5708b41-kube-api-access-lzmv9\") pod \"cert-manager-597b95cb84-6zhb5\" (UID: \"9ca237c8-f54a-420c-a115-bf62a5708b41\") " pod="cert-manager/cert-manager-597b95cb84-6zhb5"
Oct 29 22:47:22.777790 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:22.777761    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ca237c8-f54a-420c-a115-bf62a5708b41-bound-sa-token\") pod \"cert-manager-597b95cb84-6zhb5\" (UID: \"9ca237c8-f54a-420c-a115-bf62a5708b41\") " pod="cert-manager/cert-manager-597b95cb84-6zhb5"
Oct 29 22:47:22.777947 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:22.777923    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzmv9\" (UniqueName: \"kubernetes.io/projected/9ca237c8-f54a-420c-a115-bf62a5708b41-kube-api-access-lzmv9\") pod \"cert-manager-597b95cb84-6zhb5\" (UID: \"9ca237c8-f54a-420c-a115-bf62a5708b41\") " pod="cert-manager/cert-manager-597b95cb84-6zhb5"
Oct 29 22:47:22.880082 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:22.880055    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-6zhb5"
Oct 29 22:47:22.959572 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:22.959532    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-4dkx7" podUID="66774a16-0c96-482f-b7ce-f9c0d23321b9" containerName="authorino" containerID="cri-o://cb6be46c1f9063ae20e3bff8b8184c0388cd5595deecef014e92d76e6a1da053" gracePeriod=30
Oct 29 22:47:23.077766 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:23.077741    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:23.184054 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:47:23.184027    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ca237c8_f54a_420c_a115_bf62a5708b41.slice/crio-7063776f2d263b8bc43eeb3a72ba09cface62fbe7720a6ecf312e0e368aaeaf8 WatchSource:0}: Error finding container 7063776f2d263b8bc43eeb3a72ba09cface62fbe7720a6ecf312e0e368aaeaf8: Status 404 returned error can't find the container with id 7063776f2d263b8bc43eeb3a72ba09cface62fbe7720a6ecf312e0e368aaeaf8
Oct 29 22:47:23.185149 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:23.185125    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4dkx7"
Oct 29 22:47:23.185357 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:23.185338    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-6zhb5"]
Oct 29 22:47:23.249597 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:23.249565    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66774a16-0c96-482f-b7ce-f9c0d23321b9-kube-api-access-hsj9m\" (UniqueName: \"kubernetes.io/projected/66774a16-0c96-482f-b7ce-f9c0d23321b9-kube-api-access-hsj9m\") pod \"66774a16-0c96-482f-b7ce-f9c0d23321b9\" (UID: \"66774a16-0c96-482f-b7ce-f9c0d23321b9\") "
Oct 29 22:47:23.251504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:23.251478    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66774a16-0c96-482f-b7ce-f9c0d23321b9-kube-api-access-hsj9m" pod "66774a16-0c96-482f-b7ce-f9c0d23321b9" (UID: "66774a16-0c96-482f-b7ce-f9c0d23321b9"). InnerVolumeSpecName "kube-api-access-hsj9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:23.351247 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:23.351214    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsj9m\" (UniqueName: \"kubernetes.io/projected/66774a16-0c96-482f-b7ce-f9c0d23321b9-kube-api-access-hsj9m\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:47:23.965523 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:23.965487    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-6zhb5" event={"ID":"9ca237c8-f54a-420c-a115-bf62a5708b41","Type":"ContainerStarted","Data":"ea99bf776a1742beda5f341947bd74bfe75c936f941d0edd39cc2c62754ec6f7"}
Oct 29 22:47:23.965523 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:23.965527    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-6zhb5" event={"ID":"9ca237c8-f54a-420c-a115-bf62a5708b41","Type":"ContainerStarted","Data":"7063776f2d263b8bc43eeb3a72ba09cface62fbe7720a6ecf312e0e368aaeaf8"}
Oct 29 22:47:23.966846 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:23.966819    5428 generic.go:356] "Generic (PLEG): container finished" podID="66774a16-0c96-482f-b7ce-f9c0d23321b9" containerID="cb6be46c1f9063ae20e3bff8b8184c0388cd5595deecef014e92d76e6a1da053" exitCode=0
Oct 29 22:47:23.966967 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:23.966853    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4dkx7"
Oct 29 22:47:23.966967 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:23.966877    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4dkx7" event={"ID":"66774a16-0c96-482f-b7ce-f9c0d23321b9","Type":"ContainerDied","Data":"cb6be46c1f9063ae20e3bff8b8184c0388cd5595deecef014e92d76e6a1da053"}
Oct 29 22:47:23.966967 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:23.966901    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4dkx7" event={"ID":"66774a16-0c96-482f-b7ce-f9c0d23321b9","Type":"ContainerDied","Data":"21b445dcd23e24bb6d0676c46ddc3b4e719fe9e1a81193f7b149d31556a27d28"}
Oct 29 22:47:23.966967 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:23.966918    5428 scope.go:122] "RemoveContainer" containerID="cb6be46c1f9063ae20e3bff8b8184c0388cd5595deecef014e92d76e6a1da053"
Oct 29 22:47:23.976291 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:23.976274    5428 scope.go:122] "RemoveContainer" containerID="cb6be46c1f9063ae20e3bff8b8184c0388cd5595deecef014e92d76e6a1da053"
Oct 29 22:47:23.976550 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:47:23.976533    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb6be46c1f9063ae20e3bff8b8184c0388cd5595deecef014e92d76e6a1da053\": container with ID starting with cb6be46c1f9063ae20e3bff8b8184c0388cd5595deecef014e92d76e6a1da053 not found: ID does not exist" containerID="cb6be46c1f9063ae20e3bff8b8184c0388cd5595deecef014e92d76e6a1da053"
Oct 29 22:47:23.976595 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:23.976568    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb6be46c1f9063ae20e3bff8b8184c0388cd5595deecef014e92d76e6a1da053"} err="failed to get container status \"cb6be46c1f9063ae20e3bff8b8184c0388cd5595deecef014e92d76e6a1da053\": rpc error: code = NotFound desc = could not find container \"cb6be46c1f9063ae20e3bff8b8184c0388cd5595deecef014e92d76e6a1da053\": container with ID starting with cb6be46c1f9063ae20e3bff8b8184c0388cd5595deecef014e92d76e6a1da053 not found: ID does not exist"
Oct 29 22:47:24.034834 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:24.034715    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-6zhb5" podStartSLOduration=2.034700647 podStartE2EDuration="2.034700647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:24.033043364 +0000 UTC m=+247.375992113" watchObservedRunningTime="2026-10-29 22:47:24.034700647 +0000 UTC m=+247.377649391"
Oct 29 22:47:24.081956 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:24.081922    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4dkx7"]
Oct 29 22:47:24.085011 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:24.084977    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4dkx7"]
Oct 29 22:47:24.774358 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:24.774329    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66774a16-0c96-482f-b7ce-f9c0d23321b9" path="/var/lib/kubelet/pods/66774a16-0c96-482f-b7ce-f9c0d23321b9/volumes"
Oct 29 22:47:25.238784 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:25.238748    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:25.359612 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:25.359578    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:30.454687 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:30.454652    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:30.564579 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:30.564540    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:30.860884 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:30.860782    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:30.991212 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:30.991170    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:31.170702 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:31.170672    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:34.915664 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:34.915633    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-6pqtr"
Oct 29 22:47:35.168144 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:35.168042    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:35.287216 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:35.287182    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:35.916522 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:35.916484    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:38.254945 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:38.254884    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:38.374730 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:38.374694    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:40.489391 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:40.489355    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:40.613628 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:40.613595    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:43.865622 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:43.865585    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:43.974608 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:43.974563    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:46.868020 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:46.867986    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:46.988082 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:46.988037    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:47.365402 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:47:47.365364    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:47:55.255841 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:55.255788    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:55.408074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:55.408033    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:55.529090 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:55.528995    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:55.645982 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:55.645945    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:58.860784 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:58.860672    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:59.055115 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:47:59.055076    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:00.317284 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.317245    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"]
Oct 29 22:48:00.317665 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.317596    5428 cpu_manager.go:414] "containerMap: removing container" podUID="66774a16-0c96-482f-b7ce-f9c0d23321b9" containerName="authorino"
Oct 29 22:48:00.317665 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.317614    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66774a16-0c96-482f-b7ce-f9c0d23321b9" containerName="authorino"
Oct 29 22:48:00.317734 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.317714    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="66774a16-0c96-482f-b7ce-f9c0d23321b9" containerName="authorino"
Oct 29 22:48:00.320545 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.320523    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"
Oct 29 22:48:00.348138 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.348107    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 29 22:48:00.348281 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.348145    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 29 22:48:00.350575 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.350555    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-7mtsd\""
Oct 29 22:48:00.350963 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.350936    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 29 22:48:00.355473 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.355449    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"]
Oct 29 22:48:00.373995 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.373963    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/408dc1af-76b7-41fa-b262-915fcf9b4ffa-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjnhg\" (UID: \"408dc1af-76b7-41fa-b262-915fcf9b4ffa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"
Oct 29 22:48:00.374140 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.374001    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s58n\" (UniqueName: \"kubernetes.io/projected/408dc1af-76b7-41fa-b262-915fcf9b4ffa-kube-api-access-2s58n\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjnhg\" (UID: \"408dc1af-76b7-41fa-b262-915fcf9b4ffa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"
Oct 29 22:48:00.374140 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.374098    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/408dc1af-76b7-41fa-b262-915fcf9b4ffa-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjnhg\" (UID: \"408dc1af-76b7-41fa-b262-915fcf9b4ffa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"
Oct 29 22:48:00.374140 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.374130    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/408dc1af-76b7-41fa-b262-915fcf9b4ffa-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjnhg\" (UID: \"408dc1af-76b7-41fa-b262-915fcf9b4ffa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"
Oct 29 22:48:00.475361 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.475327    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/408dc1af-76b7-41fa-b262-915fcf9b4ffa-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjnhg\" (UID: \"408dc1af-76b7-41fa-b262-915fcf9b4ffa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"
Oct 29 22:48:00.475531 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.475371    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s58n\" (UniqueName: \"kubernetes.io/projected/408dc1af-76b7-41fa-b262-915fcf9b4ffa-kube-api-access-2s58n\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjnhg\" (UID: \"408dc1af-76b7-41fa-b262-915fcf9b4ffa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"
Oct 29 22:48:00.475531 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.475409    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/408dc1af-76b7-41fa-b262-915fcf9b4ffa-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjnhg\" (UID: \"408dc1af-76b7-41fa-b262-915fcf9b4ffa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"
Oct 29 22:48:00.475531 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.475434    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/408dc1af-76b7-41fa-b262-915fcf9b4ffa-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjnhg\" (UID: \"408dc1af-76b7-41fa-b262-915fcf9b4ffa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"
Oct 29 22:48:00.475531 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:48:00.475465    5428 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Oct 29 22:48:00.475696 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:48:00.475562    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/408dc1af-76b7-41fa-b262-915fcf9b4ffa-proxy-source podName:408dc1af-76b7-41fa-b262-915fcf9b4ffa nodeName:}" failed. No retries permitted until 2026-10-29 22:48:00.975542646 +0000 UTC m=+284.318491390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/408dc1af-76b7-41fa-b262-915fcf9b4ffa-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-vjnhg" (UID: "408dc1af-76b7-41fa-b262-915fcf9b4ffa") : configmap "usage-tenancy-proxy-source" not found
Oct 29 22:48:00.475696 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:48:00.475584    5428 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Oct 29 22:48:00.475696 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:48:00.475678    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/408dc1af-76b7-41fa-b262-915fcf9b4ffa-tls-cert podName:408dc1af-76b7-41fa-b262-915fcf9b4ffa nodeName:}" failed. No retries permitted until 2026-10-29 22:48:00.975661504 +0000 UTC m=+284.318610243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/408dc1af-76b7-41fa-b262-915fcf9b4ffa-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-vjnhg" (UID: "408dc1af-76b7-41fa-b262-915fcf9b4ffa") : secret "usage-logs-tenancy-proxy-tls" not found
Oct 29 22:48:00.476089 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.476073    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/408dc1af-76b7-41fa-b262-915fcf9b4ffa-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjnhg\" (UID: \"408dc1af-76b7-41fa-b262-915fcf9b4ffa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"
Oct 29 22:48:00.507715 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.507689    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s58n\" (UniqueName: \"kubernetes.io/projected/408dc1af-76b7-41fa-b262-915fcf9b4ffa-kube-api-access-2s58n\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjnhg\" (UID: \"408dc1af-76b7-41fa-b262-915fcf9b4ffa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"
Oct 29 22:48:00.555773 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.555740    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:00.675057 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.675026    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:00.979860 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.979758    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/408dc1af-76b7-41fa-b262-915fcf9b4ffa-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjnhg\" (UID: \"408dc1af-76b7-41fa-b262-915fcf9b4ffa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"
Oct 29 22:48:00.980009 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.979862    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/408dc1af-76b7-41fa-b262-915fcf9b4ffa-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjnhg\" (UID: \"408dc1af-76b7-41fa-b262-915fcf9b4ffa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"
Oct 29 22:48:00.980669 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.980648    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/408dc1af-76b7-41fa-b262-915fcf9b4ffa-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjnhg\" (UID: \"408dc1af-76b7-41fa-b262-915fcf9b4ffa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"
Oct 29 22:48:00.982284 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:00.982261    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/408dc1af-76b7-41fa-b262-915fcf9b4ffa-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vjnhg\" (UID: \"408dc1af-76b7-41fa-b262-915fcf9b4ffa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"
Oct 29 22:48:01.233275 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:01.233193    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"
Oct 29 22:48:01.565715 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:01.565633    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"]
Oct 29 22:48:01.569624 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:48:01.569590    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408dc1af_76b7_41fa_b262_915fcf9b4ffa.slice/crio-fe24e21130a9b2773fb12cf00b35a4d0516c124a3a086b1667c2ed1a8599a5b9 WatchSource:0}: Error finding container fe24e21130a9b2773fb12cf00b35a4d0516c124a3a086b1667c2ed1a8599a5b9: Status 404 returned error can't find the container with id fe24e21130a9b2773fb12cf00b35a4d0516c124a3a086b1667c2ed1a8599a5b9
Oct 29 22:48:02.141629 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:02.141595    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg" event={"ID":"408dc1af-76b7-41fa-b262-915fcf9b4ffa","Type":"ContainerStarted","Data":"fe24e21130a9b2773fb12cf00b35a4d0516c124a3a086b1667c2ed1a8599a5b9"}
Oct 29 22:48:06.423466 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.423396    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:06.546088 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.546047    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:06.655411 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.655332    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:06.690440 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.690349    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lz8kp"]
Oct 29 22:48:06.716339 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.716307    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lz8kp"]
Oct 29 22:48:06.716517 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.716432    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-lz8kp"
Oct 29 22:48:06.732840 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.731327    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 29 22:48:06.733997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.733970    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz"]
Oct 29 22:48:06.735752 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.735718    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld2rc\" (UniqueName: \"kubernetes.io/projected/1ad543b2-b749-40f3-9210-4026141bbf3d-kube-api-access-ld2rc\") pod \"kserve-controller-manager-776cc66d47-lz8kp\" (UID: \"1ad543b2-b749-40f3-9210-4026141bbf3d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lz8kp"
Oct 29 22:48:06.735895 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.735791    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ad543b2-b749-40f3-9210-4026141bbf3d-cert\") pod \"kserve-controller-manager-776cc66d47-lz8kp\" (UID: \"1ad543b2-b749-40f3-9210-4026141bbf3d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lz8kp"
Oct 29 22:48:06.739114 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.739088    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz"
Oct 29 22:48:06.741761 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.741729    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wqpxn\""
Oct 29 22:48:06.755655 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.755634    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jv9jd\""
Oct 29 22:48:06.756010 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.755989    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 29 22:48:06.764643 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.764616    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz"]
Oct 29 22:48:06.782045 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.782008    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-5k8zg"]
Oct 29 22:48:06.785464 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.785399    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:06.787020 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.786325    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-5k8zg"
Oct 29 22:48:06.810654 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.810625    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-5k8zg"]
Oct 29 22:48:06.819855 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.817883    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-rpc87\""
Oct 29 22:48:06.819855 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.817911    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 29 22:48:06.838386 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.831884    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-86vff"]
Oct 29 22:48:06.838386 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.837027    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-86vff"
Oct 29 22:48:06.838386 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.837055    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld2rc\" (UniqueName: \"kubernetes.io/projected/1ad543b2-b749-40f3-9210-4026141bbf3d-kube-api-access-ld2rc\") pod \"kserve-controller-manager-776cc66d47-lz8kp\" (UID: \"1ad543b2-b749-40f3-9210-4026141bbf3d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lz8kp"
Oct 29 22:48:06.838386 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.837099    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf6c054d-59e2-43d6-bad1-ee8c54690e83-tls-certs\") pod \"model-serving-api-75bd89fdfc-5k8zg\" (UID: \"bf6c054d-59e2-43d6-bad1-ee8c54690e83\") " pod="opendatahub/model-serving-api-75bd89fdfc-5k8zg"
Oct 29 22:48:06.838386 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.837160    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np8gd\" (UniqueName: \"kubernetes.io/projected/6b0118cf-900e-4aa2-891d-9bfb4e0c6db4-kube-api-access-np8gd\") pod \"llmisvc-controller-manager-869d5dcc5d-s7xtz\" (UID: \"6b0118cf-900e-4aa2-891d-9bfb4e0c6db4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz"
Oct 29 22:48:06.838386 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.837239    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p59lr\" (UniqueName: \"kubernetes.io/projected/bf6c054d-59e2-43d6-bad1-ee8c54690e83-kube-api-access-p59lr\") pod \"model-serving-api-75bd89fdfc-5k8zg\" (UID: \"bf6c054d-59e2-43d6-bad1-ee8c54690e83\") " pod="opendatahub/model-serving-api-75bd89fdfc-5k8zg"
Oct 29 22:48:06.838386 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.837280    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ad543b2-b749-40f3-9210-4026141bbf3d-cert\") pod \"kserve-controller-manager-776cc66d47-lz8kp\" (UID: \"1ad543b2-b749-40f3-9210-4026141bbf3d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lz8kp"
Oct 29 22:48:06.838386 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.837356    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b0118cf-900e-4aa2-891d-9bfb4e0c6db4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-s7xtz\" (UID: \"6b0118cf-900e-4aa2-891d-9bfb4e0c6db4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz"
Oct 29 22:48:06.840487 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.840465    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ad543b2-b749-40f3-9210-4026141bbf3d-cert\") pod \"kserve-controller-manager-776cc66d47-lz8kp\" (UID: \"1ad543b2-b749-40f3-9210-4026141bbf3d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lz8kp"
Oct 29 22:48:06.852277 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.852254    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 29 22:48:06.856981 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.856943    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-cz6cn\""
Oct 29 22:48:06.857079 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.857020    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 29 22:48:06.859321 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.859295    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-86vff"]
Oct 29 22:48:06.870732 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.870711    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld2rc\" (UniqueName: \"kubernetes.io/projected/1ad543b2-b749-40f3-9210-4026141bbf3d-kube-api-access-ld2rc\") pod \"kserve-controller-manager-776cc66d47-lz8kp\" (UID: \"1ad543b2-b749-40f3-9210-4026141bbf3d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lz8kp"
Oct 29 22:48:06.897661 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.897617    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:06.938438 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.938406    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf6c054d-59e2-43d6-bad1-ee8c54690e83-tls-certs\") pod \"model-serving-api-75bd89fdfc-5k8zg\" (UID: \"bf6c054d-59e2-43d6-bad1-ee8c54690e83\") " pod="opendatahub/model-serving-api-75bd89fdfc-5k8zg"
Oct 29 22:48:06.938597 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.938445    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np8gd\" (UniqueName: \"kubernetes.io/projected/6b0118cf-900e-4aa2-891d-9bfb4e0c6db4-kube-api-access-np8gd\") pod \"llmisvc-controller-manager-869d5dcc5d-s7xtz\" (UID: \"6b0118cf-900e-4aa2-891d-9bfb4e0c6db4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz"
Oct 29 22:48:06.938597 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.938469    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p59lr\" (UniqueName: \"kubernetes.io/projected/bf6c054d-59e2-43d6-bad1-ee8c54690e83-kube-api-access-p59lr\") pod \"model-serving-api-75bd89fdfc-5k8zg\" (UID: \"bf6c054d-59e2-43d6-bad1-ee8c54690e83\") " pod="opendatahub/model-serving-api-75bd89fdfc-5k8zg"
Oct 29 22:48:06.938597 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.938494    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b0118cf-900e-4aa2-891d-9bfb4e0c6db4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-s7xtz\" (UID: \"6b0118cf-900e-4aa2-891d-9bfb4e0c6db4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz"
Oct 29 22:48:06.938597 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.938515    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27d64fc8-f24c-433d-a320-9d272d87bd8d-cert\") pod \"odh-model-controller-769875db56-86vff\" (UID: \"27d64fc8-f24c-433d-a320-9d272d87bd8d\") " pod="opendatahub/odh-model-controller-769875db56-86vff"
Oct 29 22:48:06.938597 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.938532    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vnnv\" (UniqueName: \"kubernetes.io/projected/27d64fc8-f24c-433d-a320-9d272d87bd8d-kube-api-access-5vnnv\") pod \"odh-model-controller-769875db56-86vff\" (UID: \"27d64fc8-f24c-433d-a320-9d272d87bd8d\") " pod="opendatahub/odh-model-controller-769875db56-86vff"
Oct 29 22:48:06.938597 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.938554    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d64fc8-f24c-433d-a320-9d272d87bd8d-metrics-certs\") pod \"odh-model-controller-769875db56-86vff\" (UID: \"27d64fc8-f24c-433d-a320-9d272d87bd8d\") " pod="opendatahub/odh-model-controller-769875db56-86vff"
Oct 29 22:48:06.942294 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.941562    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf6c054d-59e2-43d6-bad1-ee8c54690e83-tls-certs\") pod \"model-serving-api-75bd89fdfc-5k8zg\" (UID: \"bf6c054d-59e2-43d6-bad1-ee8c54690e83\") " pod="opendatahub/model-serving-api-75bd89fdfc-5k8zg"
Oct 29 22:48:06.942294 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.942023    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b0118cf-900e-4aa2-891d-9bfb4e0c6db4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-s7xtz\" (UID: \"6b0118cf-900e-4aa2-891d-9bfb4e0c6db4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz"
Oct 29 22:48:06.960794 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.960757    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np8gd\" (UniqueName: \"kubernetes.io/projected/6b0118cf-900e-4aa2-891d-9bfb4e0c6db4-kube-api-access-np8gd\") pod \"llmisvc-controller-manager-869d5dcc5d-s7xtz\" (UID: \"6b0118cf-900e-4aa2-891d-9bfb4e0c6db4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz"
Oct 29 22:48:06.961889 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:06.961868    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p59lr\" (UniqueName: \"kubernetes.io/projected/bf6c054d-59e2-43d6-bad1-ee8c54690e83-kube-api-access-p59lr\") pod \"model-serving-api-75bd89fdfc-5k8zg\" (UID: \"bf6c054d-59e2-43d6-bad1-ee8c54690e83\") " pod="opendatahub/model-serving-api-75bd89fdfc-5k8zg"
Oct 29 22:48:07.038652 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:07.038612    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-lz8kp"
Oct 29 22:48:07.039090 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:07.039060    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27d64fc8-f24c-433d-a320-9d272d87bd8d-cert\") pod \"odh-model-controller-769875db56-86vff\" (UID: \"27d64fc8-f24c-433d-a320-9d272d87bd8d\") " pod="opendatahub/odh-model-controller-769875db56-86vff"
Oct 29 22:48:07.039191 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:07.039119    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vnnv\" (UniqueName: \"kubernetes.io/projected/27d64fc8-f24c-433d-a320-9d272d87bd8d-kube-api-access-5vnnv\") pod \"odh-model-controller-769875db56-86vff\" (UID: \"27d64fc8-f24c-433d-a320-9d272d87bd8d\") " pod="opendatahub/odh-model-controller-769875db56-86vff"
Oct 29 22:48:07.039191 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:07.039161    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d64fc8-f24c-433d-a320-9d272d87bd8d-metrics-certs\") pod \"odh-model-controller-769875db56-86vff\" (UID: \"27d64fc8-f24c-433d-a320-9d272d87bd8d\") " pod="opendatahub/odh-model-controller-769875db56-86vff"
Oct 29 22:48:07.042093 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:07.042063    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27d64fc8-f24c-433d-a320-9d272d87bd8d-cert\") pod \"odh-model-controller-769875db56-86vff\" (UID: \"27d64fc8-f24c-433d-a320-9d272d87bd8d\") " pod="opendatahub/odh-model-controller-769875db56-86vff"
Oct 29 22:48:07.042212 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:07.042184    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d64fc8-f24c-433d-a320-9d272d87bd8d-metrics-certs\") pod \"odh-model-controller-769875db56-86vff\" (UID: \"27d64fc8-f24c-433d-a320-9d272d87bd8d\") " pod="opendatahub/odh-model-controller-769875db56-86vff"
Oct 29 22:48:07.058499 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:07.058467    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz"
Oct 29 22:48:07.062502 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:07.062255    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vnnv\" (UniqueName: \"kubernetes.io/projected/27d64fc8-f24c-433d-a320-9d272d87bd8d-kube-api-access-5vnnv\") pod \"odh-model-controller-769875db56-86vff\" (UID: \"27d64fc8-f24c-433d-a320-9d272d87bd8d\") " pod="opendatahub/odh-model-controller-769875db56-86vff"
Oct 29 22:48:07.102367 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:07.102339    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-5k8zg"
Oct 29 22:48:07.166600 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:07.166571    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-86vff"
Oct 29 22:48:07.189463 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:07.189425    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:07.354352 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:07.354322    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lz8kp"]
Oct 29 22:48:07.358300 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:48:07.358264    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad543b2_b749_40f3_9210_4026141bbf3d.slice/crio-6d522d285d9e680f14817341990aa02c910743022ec392d2c754e2447b8b579d WatchSource:0}: Error finding container 6d522d285d9e680f14817341990aa02c910743022ec392d2c754e2447b8b579d: Status 404 returned error can't find the container with id 6d522d285d9e680f14817341990aa02c910743022ec392d2c754e2447b8b579d
Oct 29 22:48:07.398608 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:07.398550    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz"]
Oct 29 22:48:07.401732 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:48:07.401697    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6b0118cf_900e_4aa2_891d_9bfb4e0c6db4.slice/crio-6a71b9c8ea3d16d6e4cf7499cc2b05dd5fad566ce67addc30e8152a6ae45be6a WatchSource:0}: Error finding container 6a71b9c8ea3d16d6e4cf7499cc2b05dd5fad566ce67addc30e8152a6ae45be6a: Status 404 returned error can't find the container with id 6a71b9c8ea3d16d6e4cf7499cc2b05dd5fad566ce67addc30e8152a6ae45be6a
Oct 29 22:48:07.438374 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:07.438350    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-5k8zg"]
Oct 29 22:48:07.443501 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:48:07.441916    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6c054d_59e2_43d6_bad1_ee8c54690e83.slice/crio-6ad24423ff5e6851926ccde1992f7a065527e1080b0dbeb95c32c036a8a3b34e WatchSource:0}: Error finding container 6ad24423ff5e6851926ccde1992f7a065527e1080b0dbeb95c32c036a8a3b34e: Status 404 returned error can't find the container with id 6ad24423ff5e6851926ccde1992f7a065527e1080b0dbeb95c32c036a8a3b34e
Oct 29 22:48:07.491278 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:07.491200    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-86vff"]
Oct 29 22:48:07.495386 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:48:07.495363    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d64fc8_f24c_433d_a320_9d272d87bd8d.slice/crio-869b592a97545ca14df9e38ac2c0dd514a1381dfa93844e38f3d3856fac54366 WatchSource:0}: Error finding container 869b592a97545ca14df9e38ac2c0dd514a1381dfa93844e38f3d3856fac54366: Status 404 returned error can't find the container with id 869b592a97545ca14df9e38ac2c0dd514a1381dfa93844e38f3d3856fac54366
Oct 29 22:48:08.184761 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:08.184621    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-5k8zg" event={"ID":"bf6c054d-59e2-43d6-bad1-ee8c54690e83","Type":"ContainerStarted","Data":"6ad24423ff5e6851926ccde1992f7a065527e1080b0dbeb95c32c036a8a3b34e"}
Oct 29 22:48:08.194625 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:08.194559    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lz8kp" event={"ID":"1ad543b2-b749-40f3-9210-4026141bbf3d","Type":"ContainerStarted","Data":"6d522d285d9e680f14817341990aa02c910743022ec392d2c754e2447b8b579d"}
Oct 29 22:48:08.200886 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:08.200785    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-86vff" event={"ID":"27d64fc8-f24c-433d-a320-9d272d87bd8d","Type":"ContainerStarted","Data":"869b592a97545ca14df9e38ac2c0dd514a1381dfa93844e38f3d3856fac54366"}
Oct 29 22:48:08.203942 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:08.203910    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz" event={"ID":"6b0118cf-900e-4aa2-891d-9bfb4e0c6db4","Type":"ContainerStarted","Data":"6a71b9c8ea3d16d6e4cf7499cc2b05dd5fad566ce67addc30e8152a6ae45be6a"}
Oct 29 22:48:09.100508 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:09.100469    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:09.211669 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:09.211560    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:09.331749 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:09.331671    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:09.444347 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:09.444314    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:09.577962 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:09.577897    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:11.227701 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:11.227554    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-5k8zg" event={"ID":"bf6c054d-59e2-43d6-bad1-ee8c54690e83","Type":"ContainerStarted","Data":"9e5c9465663829d1d587152159725e8c50e423ef12d444c57aceac14204387c7"}
Oct 29 22:48:11.227701 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:11.227666    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-5k8zg"
Oct 29 22:48:11.230070 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:11.230039    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lz8kp" event={"ID":"1ad543b2-b749-40f3-9210-4026141bbf3d","Type":"ContainerStarted","Data":"185678ff5fdd653e95ea6b9baddc022d3c2dd7533c0a702d4a3a4b9d44f65b84"}
Oct 29 22:48:11.230580 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:11.230556    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lz8kp"
Oct 29 22:48:11.232374 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:11.232347    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz" event={"ID":"6b0118cf-900e-4aa2-891d-9bfb4e0c6db4","Type":"ContainerStarted","Data":"b3bfbf4965f3588d1f5acec406251ba5da34e7ad699b7b231e03b0824aa13504"}
Oct 29 22:48:11.232813 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:11.232788    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz"
Oct 29 22:48:11.271045 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:11.270982    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-5k8zg" podStartSLOduration=2.254183793 podStartE2EDuration="5.27096829s" totalImagesPullingTime="3.016784497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:11.26904924 +0000 UTC m=+294.611998015" watchObservedRunningTime="2026-10-29 22:48:11.27096829 +0000 UTC m=+294.613917035"
Oct 29 22:48:11.306551 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:11.306498    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz" podStartSLOduration=1.8860815720000002 podStartE2EDuration="5.306481609s" totalImagesPullingTime="3.420400037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:11.305983179 +0000 UTC m=+294.648931926" watchObservedRunningTime="2026-10-29 22:48:11.306481609 +0000 UTC m=+294.649430357"
Oct 29 22:48:11.344550 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:11.344497    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-lz8kp" podStartSLOduration=1.784524433 podStartE2EDuration="5.344481018s" totalImagesPullingTime="3.559956585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:11.343005621 +0000 UTC m=+294.685954367" watchObservedRunningTime="2026-10-29 22:48:11.344481018 +0000 UTC m=+294.687429761"
Oct 29 22:48:12.241529 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:12.241493    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-86vff" event={"ID":"27d64fc8-f24c-433d-a320-9d272d87bd8d","Type":"ContainerStarted","Data":"3ea5d4169ffea78ef33534cce6ccb9ad40ec65113de5fc95cc99b2a6b1bfdf83"}
Oct 29 22:48:12.242408 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:12.242381    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-86vff"
Oct 29 22:48:12.280615 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:12.280559    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-86vff" podStartSLOduration=2.186929595 podStartE2EDuration="6.280541533s" totalImagesPullingTime="4.093611938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:12.278329897 +0000 UTC m=+295.621278641" watchObservedRunningTime="2026-10-29 22:48:12.280541533 +0000 UTC m=+295.623490279"
Oct 29 22:48:15.264670 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:15.264577    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg" event={"ID":"408dc1af-76b7-41fa-b262-915fcf9b4ffa","Type":"ContainerStarted","Data":"bbf51ec2d485ce8a7f6da43881049050f8258723c930cbaf672cade948ae7b43"}
Oct 29 22:48:15.265060 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:15.264915    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"
Oct 29 22:48:15.308111 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:15.308065    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg" podStartSLOduration=1.907309382 podStartE2EDuration="15.308051131s" totalImagesPullingTime="13.400741749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:15.307227784 +0000 UTC m=+298.650176541" watchObservedRunningTime="2026-10-29 22:48:15.308051131 +0000 UTC m=+298.650999876"
Oct 29 22:48:17.367339 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:48:17.367298    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:48:22.246074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:22.246043    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-5k8zg"
Oct 29 22:48:23.223056 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:23.223024    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:24.259500 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:24.259473    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-86vff"
Oct 29 22:48:26.274296 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:26.274262    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vjnhg"
Oct 29 22:48:29.406857 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:29.406793    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:29.534835 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:29.534787    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:30.843485 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:30.843448    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:30.964066 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:30.964037    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:31.083354 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:31.083322    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:40.802633 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:40.802597    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:41.813846 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:41.813813    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:42.095628 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.095534    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:42.133924 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.133894    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-gstjp"]
Oct 29 22:48:42.136302 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.136284    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-gstjp"
Oct 29 22:48:42.149243 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.149220    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xcpgt\""
Oct 29 22:48:42.160996 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.160970    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-gstjp"]
Oct 29 22:48:42.168643 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.168619    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tcr2\" (UniqueName: \"kubernetes.io/projected/91e1da77-2d35-4be6-ace8-c607ab19dccb-kube-api-access-8tcr2\") pod \"authorino-6f6944bcdd-gstjp\" (UID: \"91e1da77-2d35-4be6-ace8-c607ab19dccb\") " pod="kuadrant-system/authorino-6f6944bcdd-gstjp"
Oct 29 22:48:42.270082 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.270042    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tcr2\" (UniqueName: \"kubernetes.io/projected/91e1da77-2d35-4be6-ace8-c607ab19dccb-kube-api-access-8tcr2\") pod \"authorino-6f6944bcdd-gstjp\" (UID: \"91e1da77-2d35-4be6-ace8-c607ab19dccb\") " pod="kuadrant-system/authorino-6f6944bcdd-gstjp"
Oct 29 22:48:42.293816 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.293772    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tcr2\" (UniqueName: \"kubernetes.io/projected/91e1da77-2d35-4be6-ace8-c607ab19dccb-kube-api-access-8tcr2\") pod \"authorino-6f6944bcdd-gstjp\" (UID: \"91e1da77-2d35-4be6-ace8-c607ab19dccb\") " pod="kuadrant-system/authorino-6f6944bcdd-gstjp"
Oct 29 22:48:42.314112 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.314066    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-lbp8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": read tcp 10.133.0.2:47752->10.133.0.21:15021: read: connection reset by peer" start-of-body=
Oct 29 22:48:42.314253 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.314157    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m" podUID="fbea7545-5d53-4ba1-bb53-584acf2a2bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": read tcp 10.133.0.2:47752->10.133.0.21:15021: read: connection reset by peer"
Oct 29 22:48:42.329476 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.329449    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-gstjp"]
Oct 29 22:48:42.329666 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.329654    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-gstjp"
Oct 29 22:48:42.353602 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.353546    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-x75b6"]
Oct 29 22:48:42.356398 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.356379    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-x75b6"
Oct 29 22:48:42.370053 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.370029    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 29 22:48:42.370738 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.370711    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnjvg\" (UniqueName: \"kubernetes.io/projected/b63f69a8-beb2-4931-90e8-1e67f7a58c39-kube-api-access-rnjvg\") pod \"authorino-6889cbd8c4-x75b6\" (UID: \"b63f69a8-beb2-4931-90e8-1e67f7a58c39\") " pod="kuadrant-system/authorino-6889cbd8c4-x75b6"
Oct 29 22:48:42.370917 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.370753    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b63f69a8-beb2-4931-90e8-1e67f7a58c39-tls-cert\") pod \"authorino-6889cbd8c4-x75b6\" (UID: \"b63f69a8-beb2-4931-90e8-1e67f7a58c39\") " pod="kuadrant-system/authorino-6889cbd8c4-x75b6"
Oct 29 22:48:42.383006 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.381867    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-x75b6"]
Oct 29 22:48:42.416897 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.416865    5428 generic.go:356] "Generic (PLEG): container finished" podID="fbea7545-5d53-4ba1-bb53-584acf2a2bc7" containerID="c11fd48fad10673ac1337687063ab5412cda3e6c09e78e7f5959ed829bfc633c" exitCode=0
Oct 29 22:48:42.417045 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.416935    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m" event={"ID":"fbea7545-5d53-4ba1-bb53-584acf2a2bc7","Type":"ContainerDied","Data":"c11fd48fad10673ac1337687063ab5412cda3e6c09e78e7f5959ed829bfc633c"}
Oct 29 22:48:42.417302 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.417287    5428 scope.go:122] "RemoveContainer" containerID="c11fd48fad10673ac1337687063ab5412cda3e6c09e78e7f5959ed829bfc633c"
Oct 29 22:48:42.418500 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.418465    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:48:42.418587 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.418558    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:48:42.418629 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.418609    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:48:42.472156 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.472129    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnjvg\" (UniqueName: \"kubernetes.io/projected/b63f69a8-beb2-4931-90e8-1e67f7a58c39-kube-api-access-rnjvg\") pod \"authorino-6889cbd8c4-x75b6\" (UID: \"b63f69a8-beb2-4931-90e8-1e67f7a58c39\") " pod="kuadrant-system/authorino-6889cbd8c4-x75b6"
Oct 29 22:48:42.472603 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.472576    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b63f69a8-beb2-4931-90e8-1e67f7a58c39-tls-cert\") pod \"authorino-6889cbd8c4-x75b6\" (UID: \"b63f69a8-beb2-4931-90e8-1e67f7a58c39\") " pod="kuadrant-system/authorino-6889cbd8c4-x75b6"
Oct 29 22:48:42.474877 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.474858    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b63f69a8-beb2-4931-90e8-1e67f7a58c39-tls-cert\") pod \"authorino-6889cbd8c4-x75b6\" (UID: \"b63f69a8-beb2-4931-90e8-1e67f7a58c39\") " pod="kuadrant-system/authorino-6889cbd8c4-x75b6"
Oct 29 22:48:42.519461 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.519412    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnjvg\" (UniqueName: \"kubernetes.io/projected/b63f69a8-beb2-4931-90e8-1e67f7a58c39-kube-api-access-rnjvg\") pod \"authorino-6889cbd8c4-x75b6\" (UID: \"b63f69a8-beb2-4931-90e8-1e67f7a58c39\") " pod="kuadrant-system/authorino-6889cbd8c4-x75b6"
Oct 29 22:48:42.528654 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.528625    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-x75b6"]
Oct 29 22:48:42.528882 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.528866    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-x75b6"
Oct 29 22:48:42.560349 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.560313    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-584dd45b57-w6sfl"]
Oct 29 22:48:42.564227 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.564204    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-584dd45b57-w6sfl"
Oct 29 22:48:42.573649 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.573622    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbcd8\" (UniqueName: \"kubernetes.io/projected/8d860107-a315-4dde-852d-4b16c84dca10-kube-api-access-pbcd8\") pod \"authorino-584dd45b57-w6sfl\" (UID: \"8d860107-a315-4dde-852d-4b16c84dca10\") " pod="kuadrant-system/authorino-584dd45b57-w6sfl"
Oct 29 22:48:42.573769 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.573671    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8d860107-a315-4dde-852d-4b16c84dca10-tls-cert\") pod \"authorino-584dd45b57-w6sfl\" (UID: \"8d860107-a315-4dde-852d-4b16c84dca10\") " pod="kuadrant-system/authorino-584dd45b57-w6sfl"
Oct 29 22:48:42.591995 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.590887    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-584dd45b57-w6sfl"]
Oct 29 22:48:42.633686 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.633653    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-gstjp"]
Oct 29 22:48:42.638213 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:48:42.638179    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91e1da77_2d35_4be6_ace8_c607ab19dccb.slice/crio-319af6bd9b227c0ff0f409498c6da949dabea101b1533318ca0b47b4cf1b5266 WatchSource:0}: Error finding container 319af6bd9b227c0ff0f409498c6da949dabea101b1533318ca0b47b4cf1b5266: Status 404 returned error can't find the container with id 319af6bd9b227c0ff0f409498c6da949dabea101b1533318ca0b47b4cf1b5266
Oct 29 22:48:42.674114 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.674077    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbcd8\" (UniqueName: \"kubernetes.io/projected/8d860107-a315-4dde-852d-4b16c84dca10-kube-api-access-pbcd8\") pod \"authorino-584dd45b57-w6sfl\" (UID: \"8d860107-a315-4dde-852d-4b16c84dca10\") " pod="kuadrant-system/authorino-584dd45b57-w6sfl"
Oct 29 22:48:42.674228 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.674130    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8d860107-a315-4dde-852d-4b16c84dca10-tls-cert\") pod \"authorino-584dd45b57-w6sfl\" (UID: \"8d860107-a315-4dde-852d-4b16c84dca10\") " pod="kuadrant-system/authorino-584dd45b57-w6sfl"
Oct 29 22:48:42.676566 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.676540    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8d860107-a315-4dde-852d-4b16c84dca10-tls-cert\") pod \"authorino-584dd45b57-w6sfl\" (UID: \"8d860107-a315-4dde-852d-4b16c84dca10\") " pod="kuadrant-system/authorino-584dd45b57-w6sfl"
Oct 29 22:48:42.695869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.695840    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbcd8\" (UniqueName: \"kubernetes.io/projected/8d860107-a315-4dde-852d-4b16c84dca10-kube-api-access-pbcd8\") pod \"authorino-584dd45b57-w6sfl\" (UID: \"8d860107-a315-4dde-852d-4b16c84dca10\") " pod="kuadrant-system/authorino-584dd45b57-w6sfl"
Oct 29 22:48:42.844669 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.844636    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-x75b6"]
Oct 29 22:48:42.848133 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:48:42.848107    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb63f69a8_beb2_4931_90e8_1e67f7a58c39.slice/crio-09e271702446fe70fd75d87a3a4b89b337ef1b6b869c53c0116dec345ae936d7 WatchSource:0}: Error finding container 09e271702446fe70fd75d87a3a4b89b337ef1b6b869c53c0116dec345ae936d7: Status 404 returned error can't find the container with id 09e271702446fe70fd75d87a3a4b89b337ef1b6b869c53c0116dec345ae936d7
Oct 29 22:48:42.880965 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:42.879212    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-584dd45b57-w6sfl"
Oct 29 22:48:43.184705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.181843    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-584dd45b57-w6sfl"]
Oct 29 22:48:43.189248 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:48:43.189220    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d860107_a315_4dde_852d_4b16c84dca10.slice/crio-26c94076d0996344cb3c1870ee2f7d9d0e07a1f53588c40f071b061f2d149f7d WatchSource:0}: Error finding container 26c94076d0996344cb3c1870ee2f7d9d0e07a1f53588c40f071b061f2d149f7d: Status 404 returned error can't find the container with id 26c94076d0996344cb3c1870ee2f7d9d0e07a1f53588c40f071b061f2d149f7d
Oct 29 22:48:43.254007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.253982    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz"
Oct 29 22:48:43.256887 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.256865    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lz8kp"
Oct 29 22:48:43.426129 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.426080    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m" event={"ID":"fbea7545-5d53-4ba1-bb53-584acf2a2bc7","Type":"ContainerStarted","Data":"be3878570fe748bff76a3b8b29e2d923d8d09eda99236bb9eabda84179f06e5e"}
Oct 29 22:48:43.427822 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.427501    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-584dd45b57-w6sfl" event={"ID":"8d860107-a315-4dde-852d-4b16c84dca10","Type":"ContainerStarted","Data":"26c94076d0996344cb3c1870ee2f7d9d0e07a1f53588c40f071b061f2d149f7d"}
Oct 29 22:48:43.429126 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.429099    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-gstjp" event={"ID":"91e1da77-2d35-4be6-ace8-c607ab19dccb","Type":"ContainerStarted","Data":"ba2bed268d9737323d813f3110d5babda6ed34f5dcf9ecfaa5275816c5673fb8"}
Oct 29 22:48:43.429228 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.429139    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-gstjp" event={"ID":"91e1da77-2d35-4be6-ace8-c607ab19dccb","Type":"ContainerStarted","Data":"319af6bd9b227c0ff0f409498c6da949dabea101b1533318ca0b47b4cf1b5266"}
Oct 29 22:48:43.429228 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.429146    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-gstjp" podUID="91e1da77-2d35-4be6-ace8-c607ab19dccb" containerName="authorino" containerID="cri-o://ba2bed268d9737323d813f3110d5babda6ed34f5dcf9ecfaa5275816c5673fb8" gracePeriod=30
Oct 29 22:48:43.430548 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.430521    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-x75b6" event={"ID":"b63f69a8-beb2-4931-90e8-1e67f7a58c39","Type":"ContainerStarted","Data":"8560df4f103da1210152069f2ff72bed76befb9516347c8daebe426d9041708c"}
Oct 29 22:48:43.430548 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.430549    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-x75b6" event={"ID":"b63f69a8-beb2-4931-90e8-1e67f7a58c39","Type":"ContainerStarted","Data":"09e271702446fe70fd75d87a3a4b89b337ef1b6b869c53c0116dec345ae936d7"}
Oct 29 22:48:43.430731 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.430636    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-x75b6" podUID="b63f69a8-beb2-4931-90e8-1e67f7a58c39" containerName="authorino" containerID="cri-o://8560df4f103da1210152069f2ff72bed76befb9516347c8daebe426d9041708c" gracePeriod=30
Oct 29 22:48:43.509171 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.509119    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-x75b6" podStartSLOduration=1.061868246 podStartE2EDuration="1.50910129s" totalImagesPullingTime="447.233044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:43.508310256 +0000 UTC m=+326.851259001" watchObservedRunningTime="2026-10-29 22:48:43.50910129 +0000 UTC m=+326.852050036"
Oct 29 22:48:43.547470 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.547022    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-gstjp" podStartSLOduration=1.044552176 podStartE2EDuration="1.546996966s" totalImagesPullingTime="502.44479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:43.542521346 +0000 UTC m=+326.885470092" watchObservedRunningTime="2026-10-29 22:48:43.546996966 +0000 UTC m=+326.889945715"
Oct 29 22:48:43.744581 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.744553    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-x75b6"
Oct 29 22:48:43.752969 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.752949    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-gstjp"
Oct 29 22:48:43.791138 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.791108    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e1da77-2d35-4be6-ace8-c607ab19dccb-kube-api-access-8tcr2\" (UniqueName: \"kubernetes.io/projected/91e1da77-2d35-4be6-ace8-c607ab19dccb-kube-api-access-8tcr2\") pod \"91e1da77-2d35-4be6-ace8-c607ab19dccb\" (UID: \"91e1da77-2d35-4be6-ace8-c607ab19dccb\") "
Oct 29 22:48:43.791330 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.791310    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b63f69a8-beb2-4931-90e8-1e67f7a58c39-kube-api-access-rnjvg\" (UniqueName: \"kubernetes.io/projected/b63f69a8-beb2-4931-90e8-1e67f7a58c39-kube-api-access-rnjvg\") pod \"b63f69a8-beb2-4931-90e8-1e67f7a58c39\" (UID: \"b63f69a8-beb2-4931-90e8-1e67f7a58c39\") "
Oct 29 22:48:43.791403 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.791359    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b63f69a8-beb2-4931-90e8-1e67f7a58c39-tls-cert\" (UniqueName: \"kubernetes.io/secret/b63f69a8-beb2-4931-90e8-1e67f7a58c39-tls-cert\") pod \"b63f69a8-beb2-4931-90e8-1e67f7a58c39\" (UID: \"b63f69a8-beb2-4931-90e8-1e67f7a58c39\") "
Oct 29 22:48:43.793570 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.793523    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b63f69a8-beb2-4931-90e8-1e67f7a58c39-kube-api-access-rnjvg" pod "b63f69a8-beb2-4931-90e8-1e67f7a58c39" (UID: "b63f69a8-beb2-4931-90e8-1e67f7a58c39"). InnerVolumeSpecName "kube-api-access-rnjvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:43.793986 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.793949    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e1da77-2d35-4be6-ace8-c607ab19dccb-kube-api-access-8tcr2" pod "91e1da77-2d35-4be6-ace8-c607ab19dccb" (UID: "91e1da77-2d35-4be6-ace8-c607ab19dccb"). InnerVolumeSpecName "kube-api-access-8tcr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:43.802303 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.802275    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b63f69a8-beb2-4931-90e8-1e67f7a58c39-tls-cert" pod "b63f69a8-beb2-4931-90e8-1e67f7a58c39" (UID: "b63f69a8-beb2-4931-90e8-1e67f7a58c39"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:43.892422 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.892378    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tcr2\" (UniqueName: \"kubernetes.io/projected/91e1da77-2d35-4be6-ace8-c607ab19dccb-kube-api-access-8tcr2\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:48:43.892422 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.892417    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnjvg\" (UniqueName: \"kubernetes.io/projected/b63f69a8-beb2-4931-90e8-1e67f7a58c39-kube-api-access-rnjvg\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:48:43.892848 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:43.892435    5428 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b63f69a8-beb2-4931-90e8-1e67f7a58c39-tls-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:48:44.001500 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.001423    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:48:44.001642 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.001609    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-lbp8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Oct 29 22:48:44.001684 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.001665    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m" podUID="fbea7545-5d53-4ba1-bb53-584acf2a2bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Oct 29 22:48:44.437444 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.437409    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-584dd45b57-w6sfl" event={"ID":"8d860107-a315-4dde-852d-4b16c84dca10","Type":"ContainerStarted","Data":"1f1c8d7252a0e25889c55a63f9eac50fe867957a2cdf534028c3e8626e67f61d"}
Oct 29 22:48:44.438617 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.438591    5428 generic.go:356] "Generic (PLEG): container finished" podID="91e1da77-2d35-4be6-ace8-c607ab19dccb" containerID="ba2bed268d9737323d813f3110d5babda6ed34f5dcf9ecfaa5275816c5673fb8" exitCode=0
Oct 29 22:48:44.438745 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.438632    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-gstjp"
Oct 29 22:48:44.438745 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.438678    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-gstjp" event={"ID":"91e1da77-2d35-4be6-ace8-c607ab19dccb","Type":"ContainerDied","Data":"ba2bed268d9737323d813f3110d5babda6ed34f5dcf9ecfaa5275816c5673fb8"}
Oct 29 22:48:44.438745 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.438721    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-gstjp" event={"ID":"91e1da77-2d35-4be6-ace8-c607ab19dccb","Type":"ContainerDied","Data":"319af6bd9b227c0ff0f409498c6da949dabea101b1533318ca0b47b4cf1b5266"}
Oct 29 22:48:44.438745 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.438743    5428 scope.go:122] "RemoveContainer" containerID="ba2bed268d9737323d813f3110d5babda6ed34f5dcf9ecfaa5275816c5673fb8"
Oct 29 22:48:44.440040 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.439911    5428 generic.go:356] "Generic (PLEG): container finished" podID="b63f69a8-beb2-4931-90e8-1e67f7a58c39" containerID="8560df4f103da1210152069f2ff72bed76befb9516347c8daebe426d9041708c" exitCode=2
Oct 29 22:48:44.440040 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.439957    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-x75b6"
Oct 29 22:48:44.440040 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.440002    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-x75b6" event={"ID":"b63f69a8-beb2-4931-90e8-1e67f7a58c39","Type":"ContainerDied","Data":"8560df4f103da1210152069f2ff72bed76befb9516347c8daebe426d9041708c"}
Oct 29 22:48:44.440179 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.440041    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-x75b6" event={"ID":"b63f69a8-beb2-4931-90e8-1e67f7a58c39","Type":"ContainerDied","Data":"09e271702446fe70fd75d87a3a4b89b337ef1b6b869c53c0116dec345ae936d7"}
Oct 29 22:48:44.447671 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.447655    5428 scope.go:122] "RemoveContainer" containerID="ba2bed268d9737323d813f3110d5babda6ed34f5dcf9ecfaa5275816c5673fb8"
Oct 29 22:48:44.447948 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:48:44.447927    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2bed268d9737323d813f3110d5babda6ed34f5dcf9ecfaa5275816c5673fb8\": container with ID starting with ba2bed268d9737323d813f3110d5babda6ed34f5dcf9ecfaa5275816c5673fb8 not found: ID does not exist" containerID="ba2bed268d9737323d813f3110d5babda6ed34f5dcf9ecfaa5275816c5673fb8"
Oct 29 22:48:44.448038 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.447954    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2bed268d9737323d813f3110d5babda6ed34f5dcf9ecfaa5275816c5673fb8"} err="failed to get container status \"ba2bed268d9737323d813f3110d5babda6ed34f5dcf9ecfaa5275816c5673fb8\": rpc error: code = NotFound desc = could not find container \"ba2bed268d9737323d813f3110d5babda6ed34f5dcf9ecfaa5275816c5673fb8\": container with ID starting with ba2bed268d9737323d813f3110d5babda6ed34f5dcf9ecfaa5275816c5673fb8 not found: ID does not exist"
Oct 29 22:48:44.448038 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.447970    5428 scope.go:122] "RemoveContainer" containerID="8560df4f103da1210152069f2ff72bed76befb9516347c8daebe426d9041708c"
Oct 29 22:48:44.456366 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.456350    5428 scope.go:122] "RemoveContainer" containerID="8560df4f103da1210152069f2ff72bed76befb9516347c8daebe426d9041708c"
Oct 29 22:48:44.456616 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:48:44.456587    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8560df4f103da1210152069f2ff72bed76befb9516347c8daebe426d9041708c\": container with ID starting with 8560df4f103da1210152069f2ff72bed76befb9516347c8daebe426d9041708c not found: ID does not exist" containerID="8560df4f103da1210152069f2ff72bed76befb9516347c8daebe426d9041708c"
Oct 29 22:48:44.456661 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.456622    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8560df4f103da1210152069f2ff72bed76befb9516347c8daebe426d9041708c"} err="failed to get container status \"8560df4f103da1210152069f2ff72bed76befb9516347c8daebe426d9041708c\": rpc error: code = NotFound desc = could not find container \"8560df4f103da1210152069f2ff72bed76befb9516347c8daebe426d9041708c\": container with ID starting with 8560df4f103da1210152069f2ff72bed76befb9516347c8daebe426d9041708c not found: ID does not exist"
Oct 29 22:48:44.480823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.480628    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-584dd45b57-w6sfl" podStartSLOduration=2.171523621 podStartE2EDuration="2.480613964s" totalImagesPullingTime="309.090343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:44.480150857 +0000 UTC m=+327.823099626" watchObservedRunningTime="2026-10-29 22:48:44.480613964 +0000 UTC m=+327.823562709"
Oct 29 22:48:44.527032 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.527001    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-x75b6"]
Oct 29 22:48:44.530603 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.530580    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-x75b6"]
Oct 29 22:48:44.586902 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.585853    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-gstjp"]
Oct 29 22:48:44.588969 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.588944    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-gstjp"]
Oct 29 22:48:44.789657 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.789622    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e1da77-2d35-4be6-ace8-c607ab19dccb" path="/var/lib/kubelet/pods/91e1da77-2d35-4be6-ace8-c607ab19dccb/volumes"
Oct 29 22:48:44.790411 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:44.790379    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b63f69a8-beb2-4931-90e8-1e67f7a58c39" path="/var/lib/kubelet/pods/b63f69a8-beb2-4931-90e8-1e67f7a58c39/volumes"
Oct 29 22:48:45.001361 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:45.001330    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-lbp8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Oct 29 22:48:45.001750 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:45.001395    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m" podUID="fbea7545-5d53-4ba1-bb53-584acf2a2bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Oct 29 22:48:46.002174 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:46.002136    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-lbp8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Oct 29 22:48:46.002564 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:46.002211    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m" podUID="fbea7545-5d53-4ba1-bb53-584acf2a2bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Oct 29 22:48:47.001677 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:47.001643    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-lbp8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Oct 29 22:48:47.001863 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:47.001707    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m" podUID="fbea7545-5d53-4ba1-bb53-584acf2a2bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Oct 29 22:48:47.390188 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:48:47.390150    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:48:48.001731 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:48.001688    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-lbp8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Oct 29 22:48:48.001919 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:48.001751    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m" podUID="fbea7545-5d53-4ba1-bb53-584acf2a2bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Oct 29 22:48:49.005025 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:49.004988    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:48:49.005509 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:49.005422    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:48:49.006128 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:49.006104    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-lbp8m"
Oct 29 22:48:59.629194 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:59.629160    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:59.746135 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:48:59.746099    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:09.464083 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:09.464010    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:09.576904 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:09.576866    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:17.366492 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:49:17.366450    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:49:20.087648 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:20.087611    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:20.209052 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:20.208999    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:23.320099 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:23.320061    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:39.556491 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:39.556458    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz"]
Oct 29 22:49:39.556914 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:39.556667    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz" podUID="6b0118cf-900e-4aa2-891d-9bfb4e0c6db4" containerName="manager" containerID="cri-o://b3bfbf4965f3588d1f5acec406251ba5da34e7ad699b7b231e03b0824aa13504" gracePeriod=30
Oct 29 22:49:39.700388 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:39.700345    5428 generic.go:356] "Generic (PLEG): container finished" podID="6b0118cf-900e-4aa2-891d-9bfb4e0c6db4" containerID="b3bfbf4965f3588d1f5acec406251ba5da34e7ad699b7b231e03b0824aa13504" exitCode=0
Oct 29 22:49:39.700587 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:39.700425    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz" event={"ID":"6b0118cf-900e-4aa2-891d-9bfb4e0c6db4","Type":"ContainerDied","Data":"b3bfbf4965f3588d1f5acec406251ba5da34e7ad699b7b231e03b0824aa13504"}
Oct 29 22:49:39.804265 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:39.804243    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz"
Oct 29 22:49:39.897762 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:39.897736    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b0118cf-900e-4aa2-891d-9bfb4e0c6db4-kube-api-access-np8gd\" (UniqueName: \"kubernetes.io/projected/6b0118cf-900e-4aa2-891d-9bfb4e0c6db4-kube-api-access-np8gd\") pod \"6b0118cf-900e-4aa2-891d-9bfb4e0c6db4\" (UID: \"6b0118cf-900e-4aa2-891d-9bfb4e0c6db4\") "
Oct 29 22:49:39.897936 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:39.897781    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b0118cf-900e-4aa2-891d-9bfb4e0c6db4-cert\" (UniqueName: \"kubernetes.io/secret/6b0118cf-900e-4aa2-891d-9bfb4e0c6db4-cert\") pod \"6b0118cf-900e-4aa2-891d-9bfb4e0c6db4\" (UID: \"6b0118cf-900e-4aa2-891d-9bfb4e0c6db4\") "
Oct 29 22:49:39.899932 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:39.899908    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b0118cf-900e-4aa2-891d-9bfb4e0c6db4-kube-api-access-np8gd" pod "6b0118cf-900e-4aa2-891d-9bfb4e0c6db4" (UID: "6b0118cf-900e-4aa2-891d-9bfb4e0c6db4"). InnerVolumeSpecName "kube-api-access-np8gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:39.900047 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:39.900005    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b0118cf-900e-4aa2-891d-9bfb4e0c6db4-cert" pod "6b0118cf-900e-4aa2-891d-9bfb4e0c6db4" (UID: "6b0118cf-900e-4aa2-891d-9bfb4e0c6db4"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:39.998795 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:39.998761    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np8gd\" (UniqueName: \"kubernetes.io/projected/6b0118cf-900e-4aa2-891d-9bfb4e0c6db4-kube-api-access-np8gd\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:49:39.998795 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:39.998790    5428 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b0118cf-900e-4aa2-891d-9bfb4e0c6db4-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.523642 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:40.523594    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:40.706472 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:40.706437    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz"
Oct 29 22:49:40.706966 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:40.706470    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz" event={"ID":"6b0118cf-900e-4aa2-891d-9bfb4e0c6db4","Type":"ContainerDied","Data":"6a71b9c8ea3d16d6e4cf7499cc2b05dd5fad566ce67addc30e8152a6ae45be6a"}
Oct 29 22:49:40.706966 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:40.706540    5428 scope.go:122] "RemoveContainer" containerID="b3bfbf4965f3588d1f5acec406251ba5da34e7ad699b7b231e03b0824aa13504"
Oct 29 22:49:40.753580 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:40.753544    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz"]
Oct 29 22:49:40.763455 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:40.763432    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-s7xtz"]
Oct 29 22:49:40.774337 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:40.774281    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b0118cf-900e-4aa2-891d-9bfb4e0c6db4" path="/var/lib/kubelet/pods/6b0118cf-900e-4aa2-891d-9bfb4e0c6db4/volumes"
Oct 29 22:49:42.191149 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.191110    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:42.194781 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.194751    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/trustyai-service-operator-controller-manager-5997944679-kl2s2"]
Oct 29 22:49:42.195171 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.195157    5428 cpu_manager.go:414] "containerMap: removing container" podUID="6b0118cf-900e-4aa2-891d-9bfb4e0c6db4" containerName="manager"
Oct 29 22:49:42.195213 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.195172    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b0118cf-900e-4aa2-891d-9bfb4e0c6db4" containerName="manager"
Oct 29 22:49:42.195213 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.195181    5428 cpu_manager.go:414] "containerMap: removing container" podUID="91e1da77-2d35-4be6-ace8-c607ab19dccb" containerName="authorino"
Oct 29 22:49:42.195213 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.195187    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e1da77-2d35-4be6-ace8-c607ab19dccb" containerName="authorino"
Oct 29 22:49:42.195213 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.195202    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b63f69a8-beb2-4931-90e8-1e67f7a58c39" containerName="authorino"
Oct 29 22:49:42.195213 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.195208    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b63f69a8-beb2-4931-90e8-1e67f7a58c39" containerName="authorino"
Oct 29 22:49:42.195370 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.195267    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b0118cf-900e-4aa2-891d-9bfb4e0c6db4" containerName="manager"
Oct 29 22:49:42.195370 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.195277    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e1da77-2d35-4be6-ace8-c607ab19dccb" containerName="authorino"
Oct 29 22:49:42.195370 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.195284    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b63f69a8-beb2-4931-90e8-1e67f7a58c39" containerName="authorino"
Oct 29 22:49:42.199593 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.199577    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/trustyai-service-operator-controller-manager-5997944679-kl2s2"
Oct 29 22:49:42.222707 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.222681    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"trustyai-service-operator-controller-manager-dockercfg-78mkc\""
Oct 29 22:49:42.225412 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.225390    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/trustyai-service-operator-controller-manager-5997944679-kl2s2"]
Oct 29 22:49:42.308543 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.308502    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:42.319602 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.319575    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjdhp\" (UniqueName: \"kubernetes.io/projected/36e122a4-d2b7-44d4-8a45-994ea1fb5af8-kube-api-access-bjdhp\") pod \"trustyai-service-operator-controller-manager-5997944679-kl2s2\" (UID: \"36e122a4-d2b7-44d4-8a45-994ea1fb5af8\") " pod="opendatahub/trustyai-service-operator-controller-manager-5997944679-kl2s2"
Oct 29 22:49:42.420893 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.420856    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjdhp\" (UniqueName: \"kubernetes.io/projected/36e122a4-d2b7-44d4-8a45-994ea1fb5af8-kube-api-access-bjdhp\") pod \"trustyai-service-operator-controller-manager-5997944679-kl2s2\" (UID: \"36e122a4-d2b7-44d4-8a45-994ea1fb5af8\") " pod="opendatahub/trustyai-service-operator-controller-manager-5997944679-kl2s2"
Oct 29 22:49:42.442674 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.442620    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjdhp\" (UniqueName: \"kubernetes.io/projected/36e122a4-d2b7-44d4-8a45-994ea1fb5af8-kube-api-access-bjdhp\") pod \"trustyai-service-operator-controller-manager-5997944679-kl2s2\" (UID: \"36e122a4-d2b7-44d4-8a45-994ea1fb5af8\") " pod="opendatahub/trustyai-service-operator-controller-manager-5997944679-kl2s2"
Oct 29 22:49:42.511774 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.511742    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/trustyai-service-operator-controller-manager-5997944679-kl2s2"
Oct 29 22:49:42.807856 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:42.807772    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/trustyai-service-operator-controller-manager-5997944679-kl2s2"]
Oct 29 22:49:42.811946 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:49:42.811918    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36e122a4_d2b7_44d4_8a45_994ea1fb5af8.slice/crio-9e37bf64f33756699a5aa98c401cf0528354becc92c8c8f0a20ca23582cebd54 WatchSource:0}: Error finding container 9e37bf64f33756699a5aa98c401cf0528354becc92c8c8f0a20ca23582cebd54: Status 404 returned error can't find the container with id 9e37bf64f33756699a5aa98c401cf0528354becc92c8c8f0a20ca23582cebd54
Oct 29 22:49:43.724339 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:43.724300    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/trustyai-service-operator-controller-manager-5997944679-kl2s2" event={"ID":"36e122a4-d2b7-44d4-8a45-994ea1fb5af8","Type":"ContainerStarted","Data":"9e37bf64f33756699a5aa98c401cf0528354becc92c8c8f0a20ca23582cebd54"}
Oct 29 22:49:45.735940 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:45.735898    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/trustyai-service-operator-controller-manager-5997944679-kl2s2" event={"ID":"36e122a4-d2b7-44d4-8a45-994ea1fb5af8","Type":"ContainerStarted","Data":"67ebf9e1d21d1425dbd6c24d866a9c3562c3f305ae3a5875c8fb416679811bdd"}
Oct 29 22:49:45.736376 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:45.736054    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/trustyai-service-operator-controller-manager-5997944679-kl2s2"
Oct 29 22:49:45.774655 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:45.774597    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/trustyai-service-operator-controller-manager-5997944679-kl2s2" podStartSLOduration=1.468113829 podStartE2EDuration="3.774579838s" totalImagesPullingTime="2.306466009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:45.773760234 +0000 UTC m=+389.116708979" watchObservedRunningTime="2026-10-29 22:49:45.774579838 +0000 UTC m=+389.117528587"
Oct 29 22:49:47.365836 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:49:47.365784    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:49:56.742555 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:56.742509    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/trustyai-service-operator-controller-manager-5997944679-kl2s2"
Oct 29 22:49:57.712738 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:57.712701    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-qc7j9"]
Oct 29 22:49:57.716624 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:57.716602    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-qc7j9"
Oct 29 22:49:57.727092 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:57.727069    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:57.743653 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:57.743625    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 29 22:49:57.744075 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:57.743732    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 29 22:49:57.744075 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:57.743818    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-5lcdl\""
Oct 29 22:49:57.745739 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:57.745718    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-qc7j9"]
Oct 29 22:49:57.857077 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:57.857044    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gqt7\" (UniqueName: \"kubernetes.io/projected/ac0a8cc7-78e6-4760-bc60-e67950d9faa0-kube-api-access-4gqt7\") pod \"ai-gateway-controller-7d85675f98-qc7j9\" (UID: \"ac0a8cc7-78e6-4760-bc60-e67950d9faa0\") " pod="opendatahub/ai-gateway-controller-7d85675f98-qc7j9"
Oct 29 22:49:57.857230 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:57.857111    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac0a8cc7-78e6-4760-bc60-e67950d9faa0-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-qc7j9\" (UID: \"ac0a8cc7-78e6-4760-bc60-e67950d9faa0\") " pod="opendatahub/ai-gateway-controller-7d85675f98-qc7j9"
Oct 29 22:49:57.863856 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:57.863821    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:57.958537 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:57.958506    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac0a8cc7-78e6-4760-bc60-e67950d9faa0-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-qc7j9\" (UID: \"ac0a8cc7-78e6-4760-bc60-e67950d9faa0\") " pod="opendatahub/ai-gateway-controller-7d85675f98-qc7j9"
Oct 29 22:49:57.958724 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:57.958575    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gqt7\" (UniqueName: \"kubernetes.io/projected/ac0a8cc7-78e6-4760-bc60-e67950d9faa0-kube-api-access-4gqt7\") pod \"ai-gateway-controller-7d85675f98-qc7j9\" (UID: \"ac0a8cc7-78e6-4760-bc60-e67950d9faa0\") " pod="opendatahub/ai-gateway-controller-7d85675f98-qc7j9"
Oct 29 22:49:57.960902 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:57.960882    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac0a8cc7-78e6-4760-bc60-e67950d9faa0-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-qc7j9\" (UID: \"ac0a8cc7-78e6-4760-bc60-e67950d9faa0\") " pod="opendatahub/ai-gateway-controller-7d85675f98-qc7j9"
Oct 29 22:49:57.979981 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:57.979919    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gqt7\" (UniqueName: \"kubernetes.io/projected/ac0a8cc7-78e6-4760-bc60-e67950d9faa0-kube-api-access-4gqt7\") pod \"ai-gateway-controller-7d85675f98-qc7j9\" (UID: \"ac0a8cc7-78e6-4760-bc60-e67950d9faa0\") " pod="opendatahub/ai-gateway-controller-7d85675f98-qc7j9"
Oct 29 22:49:58.029647 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:58.029616    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-qc7j9"
Oct 29 22:49:58.342980 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:58.342911    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-qc7j9"]
Oct 29 22:49:58.346350 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:49:58.346317    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0a8cc7_78e6_4760_bc60_e67950d9faa0.slice/crio-e6a70a8e6aebb37423f0a85e85e564031ac78b2f3510aa22b55f7f60495fb69e WatchSource:0}: Error finding container e6a70a8e6aebb37423f0a85e85e564031ac78b2f3510aa22b55f7f60495fb69e: Status 404 returned error can't find the container with id e6a70a8e6aebb37423f0a85e85e564031ac78b2f3510aa22b55f7f60495fb69e
Oct 29 22:49:58.799496 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:49:58.799460    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-qc7j9" event={"ID":"ac0a8cc7-78e6-4760-bc60-e67950d9faa0","Type":"ContainerStarted","Data":"e6a70a8e6aebb37423f0a85e85e564031ac78b2f3510aa22b55f7f60495fb69e"}
Oct 29 22:50:00.810375 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:00.810282    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-qc7j9" event={"ID":"ac0a8cc7-78e6-4760-bc60-e67950d9faa0","Type":"ContainerStarted","Data":"b655002e0337b6b893679524a21026e9d02433af09ccb2a1cd03b1025aaebeec"}
Oct 29 22:50:00.810375 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:00.810367    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-qc7j9"
Oct 29 22:50:00.848952 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:00.848901    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-qc7j9" podStartSLOduration=1.693375071 podStartE2EDuration="3.848888979s" totalImagesPullingTime="2.155513908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:00.847012451 +0000 UTC m=+404.189961196" watchObservedRunningTime="2026-10-29 22:50:00.848888979 +0000 UTC m=+404.191837723"
Oct 29 22:50:02.606102 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:02.606063    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:02.692978 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:02.692941    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-822k7"]
Oct 29 22:50:02.696484 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:02.696467    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-822k7"
Oct 29 22:50:02.709250 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:02.709227    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 29 22:50:02.719668 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:02.719637    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jqbbq\""
Oct 29 22:50:02.721883 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:02.721860    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-822k7"]
Oct 29 22:50:02.725292 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:02.725249    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:02.759779 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:02.759746    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zv5lm"]
Oct 29 22:50:02.763175 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:02.763160    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zv5lm"
Oct 29 22:50:02.784773 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:02.784748    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zv5lm"]
Oct 29 22:50:02.803212 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:02.803174    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w44bt\" (UniqueName: \"kubernetes.io/projected/5d767414-e11d-4113-90ee-f2f0ae1588a1-kube-api-access-w44bt\") pod \"payload-processing-5f78687d5b-822k7\" (UID: \"5d767414-e11d-4113-90ee-f2f0ae1588a1\") " pod="openshift-ingress/payload-processing-5f78687d5b-822k7"
Oct 29 22:50:02.803401 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:02.803378    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d767414-e11d-4113-90ee-f2f0ae1588a1-plugins-config-volume\") pod \"payload-processing-5f78687d5b-822k7\" (UID: \"5d767414-e11d-4113-90ee-f2f0ae1588a1\") " pod="openshift-ingress/payload-processing-5f78687d5b-822k7"
Oct 29 22:50:02.904193 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:02.904153    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d767414-e11d-4113-90ee-f2f0ae1588a1-plugins-config-volume\") pod \"payload-processing-5f78687d5b-822k7\" (UID: \"5d767414-e11d-4113-90ee-f2f0ae1588a1\") " pod="openshift-ingress/payload-processing-5f78687d5b-822k7"
Oct 29 22:50:02.904408 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:02.904223    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w44bt\" (UniqueName: \"kubernetes.io/projected/5d767414-e11d-4113-90ee-f2f0ae1588a1-kube-api-access-w44bt\") pod \"payload-processing-5f78687d5b-822k7\" (UID: \"5d767414-e11d-4113-90ee-f2f0ae1588a1\") " pod="openshift-ingress/payload-processing-5f78687d5b-822k7"
Oct 29 22:50:02.904408 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:02.904276    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8712a755-0297-44aa-8c53-361ff3b51ede-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zv5lm\" (UID: \"8712a755-0297-44aa-8c53-361ff3b51ede\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zv5lm"
Oct 29 22:50:02.904408 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:02.904330    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hclg6\" (UniqueName: \"kubernetes.io/projected/8712a755-0297-44aa-8c53-361ff3b51ede-kube-api-access-hclg6\") pod \"payload-pre-processing-7b98bb678c-zv5lm\" (UID: \"8712a755-0297-44aa-8c53-361ff3b51ede\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zv5lm"
Oct 29 22:50:02.905077 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:02.905049    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d767414-e11d-4113-90ee-f2f0ae1588a1-plugins-config-volume\") pod \"payload-processing-5f78687d5b-822k7\" (UID: \"5d767414-e11d-4113-90ee-f2f0ae1588a1\") " pod="openshift-ingress/payload-processing-5f78687d5b-822k7"
Oct 29 22:50:02.925988 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:02.925961    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w44bt\" (UniqueName: \"kubernetes.io/projected/5d767414-e11d-4113-90ee-f2f0ae1588a1-kube-api-access-w44bt\") pod \"payload-processing-5f78687d5b-822k7\" (UID: \"5d767414-e11d-4113-90ee-f2f0ae1588a1\") " pod="openshift-ingress/payload-processing-5f78687d5b-822k7"
Oct 29 22:50:03.005344 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:03.005305    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8712a755-0297-44aa-8c53-361ff3b51ede-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zv5lm\" (UID: \"8712a755-0297-44aa-8c53-361ff3b51ede\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zv5lm"
Oct 29 22:50:03.005507 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:03.005354    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hclg6\" (UniqueName: \"kubernetes.io/projected/8712a755-0297-44aa-8c53-361ff3b51ede-kube-api-access-hclg6\") pod \"payload-pre-processing-7b98bb678c-zv5lm\" (UID: \"8712a755-0297-44aa-8c53-361ff3b51ede\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zv5lm"
Oct 29 22:50:03.006177 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:03.006146    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8712a755-0297-44aa-8c53-361ff3b51ede-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zv5lm\" (UID: \"8712a755-0297-44aa-8c53-361ff3b51ede\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zv5lm"
Oct 29 22:50:03.009182 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:03.009161    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-822k7"
Oct 29 22:50:03.027199 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:03.027176    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hclg6\" (UniqueName: \"kubernetes.io/projected/8712a755-0297-44aa-8c53-361ff3b51ede-kube-api-access-hclg6\") pod \"payload-pre-processing-7b98bb678c-zv5lm\" (UID: \"8712a755-0297-44aa-8c53-361ff3b51ede\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zv5lm"
Oct 29 22:50:03.076642 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:03.076608    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zv5lm"
Oct 29 22:50:03.306614 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:03.306540    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-822k7"]
Oct 29 22:50:03.309779 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:50:03.309750    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d767414_e11d_4113_90ee_f2f0ae1588a1.slice/crio-831841a3a76a084411f39f63e46c13d5a86a6b2ac0887597e748031d2807a05d WatchSource:0}: Error finding container 831841a3a76a084411f39f63e46c13d5a86a6b2ac0887597e748031d2807a05d: Status 404 returned error can't find the container with id 831841a3a76a084411f39f63e46c13d5a86a6b2ac0887597e748031d2807a05d
Oct 29 22:50:03.378324 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:03.378293    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zv5lm"]
Oct 29 22:50:03.381904 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:50:03.381879    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8712a755_0297_44aa_8c53_361ff3b51ede.slice/crio-b8bddecfc9d0da1ab87fada0019b0b9db1f4d3a476b095880bc9e8c1241d9579 WatchSource:0}: Error finding container b8bddecfc9d0da1ab87fada0019b0b9db1f4d3a476b095880bc9e8c1241d9579: Status 404 returned error can't find the container with id b8bddecfc9d0da1ab87fada0019b0b9db1f4d3a476b095880bc9e8c1241d9579
Oct 29 22:50:03.831538 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:03.831440    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zv5lm" event={"ID":"8712a755-0297-44aa-8c53-361ff3b51ede","Type":"ContainerStarted","Data":"b8bddecfc9d0da1ab87fada0019b0b9db1f4d3a476b095880bc9e8c1241d9579"}
Oct 29 22:50:03.835501 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:03.835416    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-822k7" event={"ID":"5d767414-e11d-4113-90ee-f2f0ae1588a1","Type":"ContainerStarted","Data":"831841a3a76a084411f39f63e46c13d5a86a6b2ac0887597e748031d2807a05d"}
Oct 29 22:50:05.848344 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:05.848258    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-822k7" event={"ID":"5d767414-e11d-4113-90ee-f2f0ae1588a1","Type":"ContainerStarted","Data":"68c050fa173fe55cc2c0aa87e119ed170f1904dcc4f0d968ddb3c92b3110b35d"}
Oct 29 22:50:05.848742 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:05.848458    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-822k7"
Oct 29 22:50:05.849844 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:05.849818    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zv5lm" event={"ID":"8712a755-0297-44aa-8c53-361ff3b51ede","Type":"ContainerStarted","Data":"9efda7516b4c353f54150acccaece771957332e053ad4c92370ec1332faba7d7"}
Oct 29 22:50:05.849966 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:05.849947    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zv5lm"
Oct 29 22:50:05.887137 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:05.887094    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-822k7" podStartSLOduration=1.711348272 podStartE2EDuration="3.887079966s" totalImagesPullingTime="2.175731694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:05.885408307 +0000 UTC m=+409.228357051" watchObservedRunningTime="2026-10-29 22:50:05.887079966 +0000 UTC m=+409.230028735"
Oct 29 22:50:05.926296 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:05.926241    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zv5lm" podStartSLOduration=1.790114949 podStartE2EDuration="3.926223813s" totalImagesPullingTime="2.136108864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:05.924432973 +0000 UTC m=+409.267381700" watchObservedRunningTime="2026-10-29 22:50:05.926223813 +0000 UTC m=+409.269172562"
Oct 29 22:50:11.817877 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:11.817843    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-qc7j9"
Oct 29 22:50:12.087531 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:12.087457    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7bdc847df-4dfgz"]
Oct 29 22:50:12.091160 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:12.091140    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7bdc847df-4dfgz"
Oct 29 22:50:12.105130 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:12.105104    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9zrd\" (UniqueName: \"kubernetes.io/projected/6d23d603-f91c-4a7b-85e0-8c25b786b480-kube-api-access-j9zrd\") pod \"ai-gateway-controller-7bdc847df-4dfgz\" (UID: \"6d23d603-f91c-4a7b-85e0-8c25b786b480\") " pod="opendatahub/ai-gateway-controller-7bdc847df-4dfgz"
Oct 29 22:50:12.105218 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:12.105193    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d23d603-f91c-4a7b-85e0-8c25b786b480-metrics-certs\") pod \"ai-gateway-controller-7bdc847df-4dfgz\" (UID: \"6d23d603-f91c-4a7b-85e0-8c25b786b480\") " pod="opendatahub/ai-gateway-controller-7bdc847df-4dfgz"
Oct 29 22:50:12.111571 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:12.111545    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7bdc847df-4dfgz"]
Oct 29 22:50:12.205755 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:12.205718    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9zrd\" (UniqueName: \"kubernetes.io/projected/6d23d603-f91c-4a7b-85e0-8c25b786b480-kube-api-access-j9zrd\") pod \"ai-gateway-controller-7bdc847df-4dfgz\" (UID: \"6d23d603-f91c-4a7b-85e0-8c25b786b480\") " pod="opendatahub/ai-gateway-controller-7bdc847df-4dfgz"
Oct 29 22:50:12.205950 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:12.205793    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d23d603-f91c-4a7b-85e0-8c25b786b480-metrics-certs\") pod \"ai-gateway-controller-7bdc847df-4dfgz\" (UID: \"6d23d603-f91c-4a7b-85e0-8c25b786b480\") " pod="opendatahub/ai-gateway-controller-7bdc847df-4dfgz"
Oct 29 22:50:12.208219 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:12.208192    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d23d603-f91c-4a7b-85e0-8c25b786b480-metrics-certs\") pod \"ai-gateway-controller-7bdc847df-4dfgz\" (UID: \"6d23d603-f91c-4a7b-85e0-8c25b786b480\") " pod="opendatahub/ai-gateway-controller-7bdc847df-4dfgz"
Oct 29 22:50:12.228243 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:12.228218    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9zrd\" (UniqueName: \"kubernetes.io/projected/6d23d603-f91c-4a7b-85e0-8c25b786b480-kube-api-access-j9zrd\") pod \"ai-gateway-controller-7bdc847df-4dfgz\" (UID: \"6d23d603-f91c-4a7b-85e0-8c25b786b480\") " pod="opendatahub/ai-gateway-controller-7bdc847df-4dfgz"
Oct 29 22:50:12.404458 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:12.404425    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7bdc847df-4dfgz"
Oct 29 22:50:12.703407 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:12.703326    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7bdc847df-4dfgz"]
Oct 29 22:50:12.706449 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:50:12.706413    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d23d603_f91c_4a7b_85e0_8c25b786b480.slice/crio-2098e194ae17eb9634b091f036a6a926ae38f0944f07a11b925a98dfaab6cebb WatchSource:0}: Error finding container 2098e194ae17eb9634b091f036a6a926ae38f0944f07a11b925a98dfaab6cebb: Status 404 returned error can't find the container with id 2098e194ae17eb9634b091f036a6a926ae38f0944f07a11b925a98dfaab6cebb
Oct 29 22:50:12.885580 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:12.885548    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7bdc847df-4dfgz" event={"ID":"6d23d603-f91c-4a7b-85e0-8c25b786b480","Type":"ContainerStarted","Data":"2098e194ae17eb9634b091f036a6a926ae38f0944f07a11b925a98dfaab6cebb"}
Oct 29 22:50:13.892282 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:13.892245    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7bdc847df-4dfgz" event={"ID":"6d23d603-f91c-4a7b-85e0-8c25b786b480","Type":"ContainerStarted","Data":"3b0571b2620585f530ed429c740775730de0971748231fbc59b02607b2997fd8"}
Oct 29 22:50:13.892678 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:13.892323    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7bdc847df-4dfgz"
Oct 29 22:50:13.930841 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:13.930779    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7bdc847df-4dfgz" podStartSLOduration=1.623186201 podStartE2EDuration="1.930765871s" totalImagesPullingTime="307.57967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:13.930051023 +0000 UTC m=+417.272999832" watchObservedRunningTime="2026-10-29 22:50:13.930765871 +0000 UTC m=+417.273714615"
Oct 29 22:50:16.857667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:16.857642    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-822k7"
Oct 29 22:50:16.858099 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:16.857725    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zv5lm"
Oct 29 22:50:17.364899 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:50:17.364862    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:50:23.478914 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:23.478880    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:24.898944 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:24.898911    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7bdc847df-4dfgz"
Oct 29 22:50:24.970216 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:24.970184    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-qc7j9"]
Oct 29 22:50:24.970449 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:24.970393    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-qc7j9" podUID="ac0a8cc7-78e6-4760-bc60-e67950d9faa0" containerName="manager" containerID="cri-o://b655002e0337b6b893679524a21026e9d02433af09ccb2a1cd03b1025aaebeec" gracePeriod=30
Oct 29 22:50:25.206498 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:25.206470    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-qc7j9"
Oct 29 22:50:25.312602 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:25.312566    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac0a8cc7-78e6-4760-bc60-e67950d9faa0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac0a8cc7-78e6-4760-bc60-e67950d9faa0-metrics-certs\") pod \"ac0a8cc7-78e6-4760-bc60-e67950d9faa0\" (UID: \"ac0a8cc7-78e6-4760-bc60-e67950d9faa0\") "
Oct 29 22:50:25.313392 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:25.312692    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac0a8cc7-78e6-4760-bc60-e67950d9faa0-kube-api-access-4gqt7\" (UniqueName: \"kubernetes.io/projected/ac0a8cc7-78e6-4760-bc60-e67950d9faa0-kube-api-access-4gqt7\") pod \"ac0a8cc7-78e6-4760-bc60-e67950d9faa0\" (UID: \"ac0a8cc7-78e6-4760-bc60-e67950d9faa0\") "
Oct 29 22:50:25.314732 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:25.314707    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac0a8cc7-78e6-4760-bc60-e67950d9faa0-metrics-certs" pod "ac0a8cc7-78e6-4760-bc60-e67950d9faa0" (UID: "ac0a8cc7-78e6-4760-bc60-e67950d9faa0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:50:25.314850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:25.314787    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0a8cc7-78e6-4760-bc60-e67950d9faa0-kube-api-access-4gqt7" pod "ac0a8cc7-78e6-4760-bc60-e67950d9faa0" (UID: "ac0a8cc7-78e6-4760-bc60-e67950d9faa0"). InnerVolumeSpecName "kube-api-access-4gqt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:25.414311 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:25.414278    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac0a8cc7-78e6-4760-bc60-e67950d9faa0-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:50:25.414311 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:25.414307    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gqt7\" (UniqueName: \"kubernetes.io/projected/ac0a8cc7-78e6-4760-bc60-e67950d9faa0-kube-api-access-4gqt7\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:50:25.952974 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:25.952935    5428 generic.go:356] "Generic (PLEG): container finished" podID="ac0a8cc7-78e6-4760-bc60-e67950d9faa0" containerID="b655002e0337b6b893679524a21026e9d02433af09ccb2a1cd03b1025aaebeec" exitCode=0
Oct 29 22:50:25.953392 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:25.952982    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-qc7j9"
Oct 29 22:50:25.953392 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:25.953024    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-qc7j9" event={"ID":"ac0a8cc7-78e6-4760-bc60-e67950d9faa0","Type":"ContainerDied","Data":"b655002e0337b6b893679524a21026e9d02433af09ccb2a1cd03b1025aaebeec"}
Oct 29 22:50:25.953392 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:25.953078    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-qc7j9" event={"ID":"ac0a8cc7-78e6-4760-bc60-e67950d9faa0","Type":"ContainerDied","Data":"e6a70a8e6aebb37423f0a85e85e564031ac78b2f3510aa22b55f7f60495fb69e"}
Oct 29 22:50:25.953392 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:25.953100    5428 scope.go:122] "RemoveContainer" containerID="b655002e0337b6b893679524a21026e9d02433af09ccb2a1cd03b1025aaebeec"
Oct 29 22:50:25.962440 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:25.962418    5428 scope.go:122] "RemoveContainer" containerID="b655002e0337b6b893679524a21026e9d02433af09ccb2a1cd03b1025aaebeec"
Oct 29 22:50:25.962734 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:50:25.962708    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b655002e0337b6b893679524a21026e9d02433af09ccb2a1cd03b1025aaebeec\": container with ID starting with b655002e0337b6b893679524a21026e9d02433af09ccb2a1cd03b1025aaebeec not found: ID does not exist" containerID="b655002e0337b6b893679524a21026e9d02433af09ccb2a1cd03b1025aaebeec"
Oct 29 22:50:25.962824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:25.962746    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b655002e0337b6b893679524a21026e9d02433af09ccb2a1cd03b1025aaebeec"} err="failed to get container status \"b655002e0337b6b893679524a21026e9d02433af09ccb2a1cd03b1025aaebeec\": rpc error: code = NotFound desc = could not find container \"b655002e0337b6b893679524a21026e9d02433af09ccb2a1cd03b1025aaebeec\": container with ID starting with b655002e0337b6b893679524a21026e9d02433af09ccb2a1cd03b1025aaebeec not found: ID does not exist"
Oct 29 22:50:25.999889 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:25.999863    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-qc7j9"]
Oct 29 22:50:26.005398 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:26.005377    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-qc7j9"]
Oct 29 22:50:26.588022 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:26.587992    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:26.775655 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:26.775618    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac0a8cc7-78e6-4760-bc60-e67950d9faa0" path="/var/lib/kubelet/pods/ac0a8cc7-78e6-4760-bc60-e67950d9faa0/volumes"
Oct 29 22:50:27.126964 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:27.126907    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:28.362424 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:28.362356    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:28.520238 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:28.520188    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:31.822542 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.822505    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"]
Oct 29 22:50:31.823035 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.822931    5428 cpu_manager.go:414] "containerMap: removing container" podUID="ac0a8cc7-78e6-4760-bc60-e67950d9faa0" containerName="manager"
Oct 29 22:50:31.823035 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.822944    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac0a8cc7-78e6-4760-bc60-e67950d9faa0" containerName="manager"
Oct 29 22:50:31.823153 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.823054    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac0a8cc7-78e6-4760-bc60-e67950d9faa0" containerName="manager"
Oct 29 22:50:31.830703 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.830682    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:31.853865 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.853818    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 29 22:50:31.853865 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.853842    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-28mq9\""
Oct 29 22:50:31.854093 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.853987    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 29 22:50:31.854093 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.853999    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 29 22:50:31.856690 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.856667    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"]
Oct 29 22:50:31.870334 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.870307    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtdkz\" (UniqueName: \"kubernetes.io/projected/87e30ba6-6bca-4971-844c-c9509f5b0d0b-kube-api-access-xtdkz\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc\" (UID: \"87e30ba6-6bca-4971-844c-c9509f5b0d0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:31.870430 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.870358    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e30ba6-6bca-4971-844c-c9509f5b0d0b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc\" (UID: \"87e30ba6-6bca-4971-844c-c9509f5b0d0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:31.870430 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.870417    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e30ba6-6bca-4971-844c-c9509f5b0d0b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc\" (UID: \"87e30ba6-6bca-4971-844c-c9509f5b0d0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:31.870497 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.870436    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e30ba6-6bca-4971-844c-c9509f5b0d0b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc\" (UID: \"87e30ba6-6bca-4971-844c-c9509f5b0d0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:31.870497 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.870486    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e30ba6-6bca-4971-844c-c9509f5b0d0b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc\" (UID: \"87e30ba6-6bca-4971-844c-c9509f5b0d0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:31.870562 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.870517    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e30ba6-6bca-4971-844c-c9509f5b0d0b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc\" (UID: \"87e30ba6-6bca-4971-844c-c9509f5b0d0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:31.879896 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.879866    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:31.971093 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.971057    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtdkz\" (UniqueName: \"kubernetes.io/projected/87e30ba6-6bca-4971-844c-c9509f5b0d0b-kube-api-access-xtdkz\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc\" (UID: \"87e30ba6-6bca-4971-844c-c9509f5b0d0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:31.971251 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.971138    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e30ba6-6bca-4971-844c-c9509f5b0d0b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc\" (UID: \"87e30ba6-6bca-4971-844c-c9509f5b0d0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:31.971251 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.971171    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e30ba6-6bca-4971-844c-c9509f5b0d0b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc\" (UID: \"87e30ba6-6bca-4971-844c-c9509f5b0d0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:31.971251 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.971197    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e30ba6-6bca-4971-844c-c9509f5b0d0b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc\" (UID: \"87e30ba6-6bca-4971-844c-c9509f5b0d0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:31.971251 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.971221    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e30ba6-6bca-4971-844c-c9509f5b0d0b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc\" (UID: \"87e30ba6-6bca-4971-844c-c9509f5b0d0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:31.971553 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.971280    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e30ba6-6bca-4971-844c-c9509f5b0d0b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc\" (UID: \"87e30ba6-6bca-4971-844c-c9509f5b0d0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:31.971719 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.971694    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e30ba6-6bca-4971-844c-c9509f5b0d0b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc\" (UID: \"87e30ba6-6bca-4971-844c-c9509f5b0d0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:31.971811 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.971718    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e30ba6-6bca-4971-844c-c9509f5b0d0b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc\" (UID: \"87e30ba6-6bca-4971-844c-c9509f5b0d0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:31.971811 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.971756    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e30ba6-6bca-4971-844c-c9509f5b0d0b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc\" (UID: \"87e30ba6-6bca-4971-844c-c9509f5b0d0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:31.973608 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.973581    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e30ba6-6bca-4971-844c-c9509f5b0d0b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc\" (UID: \"87e30ba6-6bca-4971-844c-c9509f5b0d0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:31.973729 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.973685    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e30ba6-6bca-4971-844c-c9509f5b0d0b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc\" (UID: \"87e30ba6-6bca-4971-844c-c9509f5b0d0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:31.993026 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:31.992994    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtdkz\" (UniqueName: \"kubernetes.io/projected/87e30ba6-6bca-4971-844c-c9509f5b0d0b-kube-api-access-xtdkz\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc\" (UID: \"87e30ba6-6bca-4971-844c-c9509f5b0d0b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:32.143992 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:32.143961    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:32.451118 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:32.451037    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"]
Oct 29 22:50:32.454142 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:50:32.454103    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e30ba6_6bca_4971_844c_c9509f5b0d0b.slice/crio-a4196b3dc684c6540339b3e772dfc115820310e434a2c8fd5582e17042f43a6f WatchSource:0}: Error finding container a4196b3dc684c6540339b3e772dfc115820310e434a2c8fd5582e17042f43a6f: Status 404 returned error can't find the container with id a4196b3dc684c6540339b3e772dfc115820310e434a2c8fd5582e17042f43a6f
Oct 29 22:50:32.994820 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:32.994720    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc" event={"ID":"87e30ba6-6bca-4971-844c-c9509f5b0d0b","Type":"ContainerStarted","Data":"a4196b3dc684c6540339b3e772dfc115820310e434a2c8fd5582e17042f43a6f"}
Oct 29 22:50:34.000829 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:34.000778    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc" event={"ID":"87e30ba6-6bca-4971-844c-c9509f5b0d0b","Type":"ContainerStarted","Data":"90cec125d2a4d731b18767c94632f8aef4c6b70836f2b2355cb75c56f365b7c3"}
Oct 29 22:50:34.041483 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:34.041429    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc" podStartSLOduration=1.68355297 podStartE2EDuration="3.041414067s" totalImagesPullingTime="1.357861097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:34.039992487 +0000 UTC m=+437.382941233" watchObservedRunningTime="2026-10-29 22:50:34.041414067 +0000 UTC m=+437.384362812"
Oct 29 22:50:34.144108 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:34.144072    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:35.937983 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:35.937947    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:37.398507 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.398474    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"]
Oct 29 22:50:37.407680 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.407647    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.420607 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.420580    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 29 22:50:37.432916 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.432890    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"]
Oct 29 22:50:37.456033 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.455987    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:37.528453 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.528403    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9ab9-9d59-4f78-a953-aa6138fb888a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2s6t\" (UID: \"cbcb9ab9-9d59-4f78-a953-aa6138fb888a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.528635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.528471    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9ab9-9d59-4f78-a953-aa6138fb888a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2s6t\" (UID: \"cbcb9ab9-9d59-4f78-a953-aa6138fb888a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.528635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.528609    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pj7c\" (UniqueName: \"kubernetes.io/projected/cbcb9ab9-9d59-4f78-a953-aa6138fb888a-kube-api-access-6pj7c\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2s6t\" (UID: \"cbcb9ab9-9d59-4f78-a953-aa6138fb888a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.528731 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.528661    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9ab9-9d59-4f78-a953-aa6138fb888a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2s6t\" (UID: \"cbcb9ab9-9d59-4f78-a953-aa6138fb888a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.528776 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.528757    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbcb9ab9-9d59-4f78-a953-aa6138fb888a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2s6t\" (UID: \"cbcb9ab9-9d59-4f78-a953-aa6138fb888a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.528871 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.528854    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9ab9-9d59-4f78-a953-aa6138fb888a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2s6t\" (UID: \"cbcb9ab9-9d59-4f78-a953-aa6138fb888a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.629956 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.629920    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pj7c\" (UniqueName: \"kubernetes.io/projected/cbcb9ab9-9d59-4f78-a953-aa6138fb888a-kube-api-access-6pj7c\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2s6t\" (UID: \"cbcb9ab9-9d59-4f78-a953-aa6138fb888a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.630151 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.629970    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9ab9-9d59-4f78-a953-aa6138fb888a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2s6t\" (UID: \"cbcb9ab9-9d59-4f78-a953-aa6138fb888a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.630151 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.630014    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbcb9ab9-9d59-4f78-a953-aa6138fb888a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2s6t\" (UID: \"cbcb9ab9-9d59-4f78-a953-aa6138fb888a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.630151 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.630069    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9ab9-9d59-4f78-a953-aa6138fb888a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2s6t\" (UID: \"cbcb9ab9-9d59-4f78-a953-aa6138fb888a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.630151 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.630135    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9ab9-9d59-4f78-a953-aa6138fb888a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2s6t\" (UID: \"cbcb9ab9-9d59-4f78-a953-aa6138fb888a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.630375 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.630171    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9ab9-9d59-4f78-a953-aa6138fb888a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2s6t\" (UID: \"cbcb9ab9-9d59-4f78-a953-aa6138fb888a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.630508 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.630470    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9ab9-9d59-4f78-a953-aa6138fb888a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2s6t\" (UID: \"cbcb9ab9-9d59-4f78-a953-aa6138fb888a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.630508 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.630492    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9ab9-9d59-4f78-a953-aa6138fb888a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2s6t\" (UID: \"cbcb9ab9-9d59-4f78-a953-aa6138fb888a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.630717 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.630626    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9ab9-9d59-4f78-a953-aa6138fb888a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2s6t\" (UID: \"cbcb9ab9-9d59-4f78-a953-aa6138fb888a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.632453 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.632431    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbcb9ab9-9d59-4f78-a953-aa6138fb888a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2s6t\" (UID: \"cbcb9ab9-9d59-4f78-a953-aa6138fb888a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.632733 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.632711    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbcb9ab9-9d59-4f78-a953-aa6138fb888a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2s6t\" (UID: \"cbcb9ab9-9d59-4f78-a953-aa6138fb888a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.651670 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.651609    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pj7c\" (UniqueName: \"kubernetes.io/projected/cbcb9ab9-9d59-4f78-a953-aa6138fb888a-kube-api-access-6pj7c\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2s6t\" (UID: \"cbcb9ab9-9d59-4f78-a953-aa6138fb888a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:37.722834 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:37.721791    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:38.049630 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:38.049538    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"]
Oct 29 22:50:38.052725 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:50:38.052700    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcb9ab9_9d59_4f78_a953_aa6138fb888a.slice/crio-a5ab794e5eb33ed9d8c7e044dbf7bd0d649ac229acc4b815ce0edbf2fd22697a WatchSource:0}: Error finding container a5ab794e5eb33ed9d8c7e044dbf7bd0d649ac229acc4b815ce0edbf2fd22697a: Status 404 returned error can't find the container with id a5ab794e5eb33ed9d8c7e044dbf7bd0d649ac229acc4b815ce0edbf2fd22697a
Oct 29 22:50:38.842364 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:38.842214    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:39.028755 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:39.028718    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t" event={"ID":"cbcb9ab9-9d59-4f78-a953-aa6138fb888a","Type":"ContainerStarted","Data":"40c00019dac909a412ff8c62073f1d03a812e3d0c1912f4e01ced97b6c4c67c4"}
Oct 29 22:50:39.028974 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:39.028771    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t" event={"ID":"cbcb9ab9-9d59-4f78-a953-aa6138fb888a","Type":"ContainerStarted","Data":"a5ab794e5eb33ed9d8c7e044dbf7bd0d649ac229acc4b815ce0edbf2fd22697a"}
Oct 29 22:50:39.071193 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:39.071121    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t" podStartSLOduration=1.852747962 podStartE2EDuration="2.071099921s" totalImagesPullingTime="218.351959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:39.067636078 +0000 UTC m=+442.410584850" watchObservedRunningTime="2026-10-29 22:50:39.071099921 +0000 UTC m=+442.414048668"
Oct 29 22:50:39.722932 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:39.722896    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:41.148415 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.148376    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"]
Oct 29 22:50:41.152643 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.152620    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.179238 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.179208    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 29 22:50:41.181412 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.181385    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"]
Oct 29 22:50:41.229221 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.229186    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:41.269634 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.269597    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7ce9ea-305c-4510-86db-2a9a74021633-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj\" (UID: \"2a7ce9ea-305c-4510-86db-2a9a74021633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.269790 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.269661    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7ce9ea-305c-4510-86db-2a9a74021633-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj\" (UID: \"2a7ce9ea-305c-4510-86db-2a9a74021633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.269790 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.269719    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7ce9ea-305c-4510-86db-2a9a74021633-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj\" (UID: \"2a7ce9ea-305c-4510-86db-2a9a74021633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.269790 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.269745    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7ce9ea-305c-4510-86db-2a9a74021633-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj\" (UID: \"2a7ce9ea-305c-4510-86db-2a9a74021633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.269790 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.269781    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7ce9ea-305c-4510-86db-2a9a74021633-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj\" (UID: \"2a7ce9ea-305c-4510-86db-2a9a74021633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.269973 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.269900    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgkj7\" (UniqueName: \"kubernetes.io/projected/2a7ce9ea-305c-4510-86db-2a9a74021633-kube-api-access-hgkj7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj\" (UID: \"2a7ce9ea-305c-4510-86db-2a9a74021633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.371493 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.371454    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7ce9ea-305c-4510-86db-2a9a74021633-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj\" (UID: \"2a7ce9ea-305c-4510-86db-2a9a74021633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.371729 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.371523    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgkj7\" (UniqueName: \"kubernetes.io/projected/2a7ce9ea-305c-4510-86db-2a9a74021633-kube-api-access-hgkj7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj\" (UID: \"2a7ce9ea-305c-4510-86db-2a9a74021633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.371729 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.371582    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7ce9ea-305c-4510-86db-2a9a74021633-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj\" (UID: \"2a7ce9ea-305c-4510-86db-2a9a74021633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.371729 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.371638    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7ce9ea-305c-4510-86db-2a9a74021633-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj\" (UID: \"2a7ce9ea-305c-4510-86db-2a9a74021633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.371729 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.371676    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7ce9ea-305c-4510-86db-2a9a74021633-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj\" (UID: \"2a7ce9ea-305c-4510-86db-2a9a74021633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.371998 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.371839    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7ce9ea-305c-4510-86db-2a9a74021633-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj\" (UID: \"2a7ce9ea-305c-4510-86db-2a9a74021633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.371998 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.371953    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7ce9ea-305c-4510-86db-2a9a74021633-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj\" (UID: \"2a7ce9ea-305c-4510-86db-2a9a74021633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.372113 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.372036    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7ce9ea-305c-4510-86db-2a9a74021633-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj\" (UID: \"2a7ce9ea-305c-4510-86db-2a9a74021633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.372588 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.372560    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7ce9ea-305c-4510-86db-2a9a74021633-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj\" (UID: \"2a7ce9ea-305c-4510-86db-2a9a74021633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.374186 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.374157    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7ce9ea-305c-4510-86db-2a9a74021633-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj\" (UID: \"2a7ce9ea-305c-4510-86db-2a9a74021633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.374910 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.374885    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7ce9ea-305c-4510-86db-2a9a74021633-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj\" (UID: \"2a7ce9ea-305c-4510-86db-2a9a74021633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.394424 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.394390    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgkj7\" (UniqueName: \"kubernetes.io/projected/2a7ce9ea-305c-4510-86db-2a9a74021633-kube-api-access-hgkj7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj\" (UID: \"2a7ce9ea-305c-4510-86db-2a9a74021633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.470880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.470767    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:41.780345 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:41.780313    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"]
Oct 29 22:50:41.783313 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:50:41.783280    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7ce9ea_305c_4510_86db_2a9a74021633.slice/crio-0248e6b220e4bcd603a486da5f2d7612e168b93638c363d14c06a2aed523a034 WatchSource:0}: Error finding container 0248e6b220e4bcd603a486da5f2d7612e168b93638c363d14c06a2aed523a034: Status 404 returned error can't find the container with id 0248e6b220e4bcd603a486da5f2d7612e168b93638c363d14c06a2aed523a034
Oct 29 22:50:42.045156 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:42.045085    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj" event={"ID":"2a7ce9ea-305c-4510-86db-2a9a74021633","Type":"ContainerStarted","Data":"0248e6b220e4bcd603a486da5f2d7612e168b93638c363d14c06a2aed523a034"}
Oct 29 22:50:42.145166 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:42.145125    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:42.157164 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:42.157137    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:42.882406 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:42.882244    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:43.051990 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:43.051955    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj" event={"ID":"2a7ce9ea-305c-4510-86db-2a9a74021633","Type":"ContainerStarted","Data":"1dc2604da9cad52a069df2b4aef1561f0660c1603878627079bbd97362385c4b"}
Oct 29 22:50:43.063726 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:43.063699    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc"
Oct 29 22:50:43.093319 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:43.093250    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj" podStartSLOduration=1.884186221 podStartE2EDuration="2.093231105s" totalImagesPullingTime="209.044884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:43.090584651 +0000 UTC m=+446.433533397" watchObservedRunningTime="2026-10-29 22:50:43.093231105 +0000 UTC m=+446.436179869"
Oct 29 22:50:43.471705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:43.471672    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:45.093833 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.093775    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"]
Oct 29 22:50:45.098340 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.098317    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.113579 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.113552    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 29 22:50:45.126180 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.126151    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"]
Oct 29 22:50:45.154746 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.154721    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:45.212253 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.212213    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4034d76-e0f2-4674-928d-657820813e88-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc\" (UID: \"c4034d76-e0f2-4674-928d-657820813e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.212426 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.212281    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4034d76-e0f2-4674-928d-657820813e88-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc\" (UID: \"c4034d76-e0f2-4674-928d-657820813e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.212426 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.212334    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4034d76-e0f2-4674-928d-657820813e88-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc\" (UID: \"c4034d76-e0f2-4674-928d-657820813e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.212508 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.212433    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4034d76-e0f2-4674-928d-657820813e88-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc\" (UID: \"c4034d76-e0f2-4674-928d-657820813e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.212542 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.212521    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtccp\" (UniqueName: \"kubernetes.io/projected/c4034d76-e0f2-4674-928d-657820813e88-kube-api-access-gtccp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc\" (UID: \"c4034d76-e0f2-4674-928d-657820813e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.212609 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.212593    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4034d76-e0f2-4674-928d-657820813e88-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc\" (UID: \"c4034d76-e0f2-4674-928d-657820813e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.313567 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.313525    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4034d76-e0f2-4674-928d-657820813e88-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc\" (UID: \"c4034d76-e0f2-4674-928d-657820813e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.313738 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.313584    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4034d76-e0f2-4674-928d-657820813e88-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc\" (UID: \"c4034d76-e0f2-4674-928d-657820813e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.313738 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.313613    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4034d76-e0f2-4674-928d-657820813e88-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc\" (UID: \"c4034d76-e0f2-4674-928d-657820813e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.313738 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.313638    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtccp\" (UniqueName: \"kubernetes.io/projected/c4034d76-e0f2-4674-928d-657820813e88-kube-api-access-gtccp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc\" (UID: \"c4034d76-e0f2-4674-928d-657820813e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.313738 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.313672    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4034d76-e0f2-4674-928d-657820813e88-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc\" (UID: \"c4034d76-e0f2-4674-928d-657820813e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.313973 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.313844    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4034d76-e0f2-4674-928d-657820813e88-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc\" (UID: \"c4034d76-e0f2-4674-928d-657820813e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.314029 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.313968    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4034d76-e0f2-4674-928d-657820813e88-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc\" (UID: \"c4034d76-e0f2-4674-928d-657820813e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.314086 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.314056    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4034d76-e0f2-4674-928d-657820813e88-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc\" (UID: \"c4034d76-e0f2-4674-928d-657820813e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.314163 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.314084    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4034d76-e0f2-4674-928d-657820813e88-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc\" (UID: \"c4034d76-e0f2-4674-928d-657820813e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.316278 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.316253    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4034d76-e0f2-4674-928d-657820813e88-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc\" (UID: \"c4034d76-e0f2-4674-928d-657820813e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.316384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.316348    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4034d76-e0f2-4674-928d-657820813e88-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc\" (UID: \"c4034d76-e0f2-4674-928d-657820813e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.336304 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.336280    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtccp\" (UniqueName: \"kubernetes.io/projected/c4034d76-e0f2-4674-928d-657820813e88-kube-api-access-gtccp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc\" (UID: \"c4034d76-e0f2-4674-928d-657820813e88\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.417808 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.417775    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:45.735255 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.735137    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"]
Oct 29 22:50:45.739165 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:50:45.739122    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4034d76_e0f2_4674_928d_657820813e88.slice/crio-715f34a47e9f6383d5d71b5775aa11436784d0b877f54af12aea37940baf2e13 WatchSource:0}: Error finding container 715f34a47e9f6383d5d71b5775aa11436784d0b877f54af12aea37940baf2e13: Status 404 returned error can't find the container with id 715f34a47e9f6383d5d71b5775aa11436784d0b877f54af12aea37940baf2e13
Oct 29 22:50:45.740637 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:45.740613    5428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:50:46.081355 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:46.081307    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc" event={"ID":"c4034d76-e0f2-4674-928d-657820813e88","Type":"ContainerStarted","Data":"3e9d7c43aad98f7425a985a32caff13fdb4abab8ab8dda495c0ec1a16f9c14d2"}
Oct 29 22:50:46.081531 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:46.081366    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc" event={"ID":"c4034d76-e0f2-4674-928d-657820813e88","Type":"ContainerStarted","Data":"715f34a47e9f6383d5d71b5775aa11436784d0b877f54af12aea37940baf2e13"}
Oct 29 22:50:46.123438 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:46.123388    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc" podStartSLOduration=0.918600668 podStartE2EDuration="1.123369225s" totalImagesPullingTime="204.768557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:46.121075951 +0000 UTC m=+449.464024693" watchObservedRunningTime="2026-10-29 22:50:46.123369225 +0000 UTC m=+449.466318015"
Oct 29 22:50:46.418195 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:46.418153    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:47.374874 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:50:47.374831    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:50:47.723301 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:47.723200    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:47.736550 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:47.736521    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:48.110836 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:48.110718    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2s6t"
Oct 29 22:50:49.747904 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:49.747875    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:51.471882 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:51.471836    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:51.492241 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:51.492201    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:52.132925 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:52.132893    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj"
Oct 29 22:50:55.418970 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:55.418932    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:55.431736 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:55.431705    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:50:56.195418 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:50:56.195382    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc"
Oct 29 22:51:03.197298 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.197212    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"]
Oct 29 22:51:03.201164 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.201140    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.214539 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.214513    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.214667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.214566    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42695ed6-95ae-457a-8822-8c9e303d06b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.214667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.214658    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.214760 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.214700    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.214760 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.214732    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b4rz\" (UniqueName: \"kubernetes.io/projected/42695ed6-95ae-457a-8822-8c9e303d06b4-kube-api-access-2b4rz\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.214868 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.214772    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.214922 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.214868    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.214922 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.214889    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.215025 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.214951    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.229841 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.229785    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"]
Oct 29 22:51:03.315879 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.315843    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.315879 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.315884    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.316080 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.315904    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.316080 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.315921    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.316080 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.315965    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.316080 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.316003    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42695ed6-95ae-457a-8822-8c9e303d06b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.316080 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.316029    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.316313 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.316166    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.316313 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.316231    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b4rz\" (UniqueName: \"kubernetes.io/projected/42695ed6-95ae-457a-8822-8c9e303d06b4-kube-api-access-2b4rz\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.316410 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.316307    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.316410 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.316390    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.316514 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.316427    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.316656 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.316638    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.317089 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.317063    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42695ed6-95ae-457a-8822-8c9e303d06b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.318277 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.318252    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.318445 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.318429    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.337585 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.337557    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.337724 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.337707    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b4rz\" (UniqueName: \"kubernetes.io/projected/42695ed6-95ae-457a-8822-8c9e303d06b4-kube-api-access-2b4rz\") pod \"maas-default-gateway-openshift-default-55fbd65575-fssgc\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.514467 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.514372    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:03.846284 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.846200    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"]
Oct 29 22:51:03.849217 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:51:03.849185    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42695ed6_95ae_457a_8822_8c9e303d06b4.slice/crio-93193bd0f0966b1c799bf1b87bef44938aa30bb6eccce25565ca7da59c7c001a WatchSource:0}: Error finding container 93193bd0f0966b1c799bf1b87bef44938aa30bb6eccce25565ca7da59c7c001a: Status 404 returned error can't find the container with id 93193bd0f0966b1c799bf1b87bef44938aa30bb6eccce25565ca7da59c7c001a
Oct 29 22:51:03.851468 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.851437    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:51:03.851572 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.851513    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:51:03.851572 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:03.851560    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:51:04.229566 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:04.229513    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc" event={"ID":"42695ed6-95ae-457a-8822-8c9e303d06b4","Type":"ContainerStarted","Data":"019f4c1065a6f1f532e07f6dc23d6f2ecaac57d95542f16fbf06f6e052f1d4c6"}
Oct 29 22:51:04.230075 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:04.229607    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc" event={"ID":"42695ed6-95ae-457a-8822-8c9e303d06b4","Type":"ContainerStarted","Data":"93193bd0f0966b1c799bf1b87bef44938aa30bb6eccce25565ca7da59c7c001a"}
Oct 29 22:51:04.294310 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:04.294251    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc" podStartSLOduration=1.294237404 podStartE2EDuration="1.294237404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:04.285328138 +0000 UTC m=+467.628276883" watchObservedRunningTime="2026-10-29 22:51:04.294237404 +0000 UTC m=+467.637186224"
Oct 29 22:51:04.515101 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:04.515017    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:04.516332 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:04.516307    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-fssgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 29 22:51:04.516441 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:04.516354    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc" podUID="42695ed6-95ae-457a-8822-8c9e303d06b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 29 22:51:05.515537 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:05.515501    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-fssgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 29 22:51:05.516105 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:05.515572    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc" podUID="42695ed6-95ae-457a-8822-8c9e303d06b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 29 22:51:06.515688 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:06.515655    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-fssgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 29 22:51:06.516096 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:06.515709    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc" podUID="42695ed6-95ae-457a-8822-8c9e303d06b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 29 22:51:06.585920 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:06.585883    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f48949686-sngj6"]
Oct 29 22:51:06.590018 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:06.590000    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f48949686-sngj6"
Oct 29 22:51:06.601608 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:06.601577    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f48949686-sngj6"]
Oct 29 22:51:06.649970 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:06.649937    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scslj\" (UniqueName: \"kubernetes.io/projected/96b9b7f3-d372-44ca-adfb-79546be9f85e-kube-api-access-scslj\") pod \"authorino-6f48949686-sngj6\" (UID: \"96b9b7f3-d372-44ca-adfb-79546be9f85e\") " pod="kuadrant-system/authorino-6f48949686-sngj6"
Oct 29 22:51:06.650157 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:06.650132    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/96b9b7f3-d372-44ca-adfb-79546be9f85e-tls-cert\") pod \"authorino-6f48949686-sngj6\" (UID: \"96b9b7f3-d372-44ca-adfb-79546be9f85e\") " pod="kuadrant-system/authorino-6f48949686-sngj6"
Oct 29 22:51:06.751318 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:06.751284    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/96b9b7f3-d372-44ca-adfb-79546be9f85e-tls-cert\") pod \"authorino-6f48949686-sngj6\" (UID: \"96b9b7f3-d372-44ca-adfb-79546be9f85e\") " pod="kuadrant-system/authorino-6f48949686-sngj6"
Oct 29 22:51:06.751491 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:06.751346    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scslj\" (UniqueName: \"kubernetes.io/projected/96b9b7f3-d372-44ca-adfb-79546be9f85e-kube-api-access-scslj\") pod \"authorino-6f48949686-sngj6\" (UID: \"96b9b7f3-d372-44ca-adfb-79546be9f85e\") " pod="kuadrant-system/authorino-6f48949686-sngj6"
Oct 29 22:51:06.753885 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:06.753861    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/96b9b7f3-d372-44ca-adfb-79546be9f85e-tls-cert\") pod \"authorino-6f48949686-sngj6\" (UID: \"96b9b7f3-d372-44ca-adfb-79546be9f85e\") " pod="kuadrant-system/authorino-6f48949686-sngj6"
Oct 29 22:51:06.789472 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:06.789391    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scslj\" (UniqueName: \"kubernetes.io/projected/96b9b7f3-d372-44ca-adfb-79546be9f85e-kube-api-access-scslj\") pod \"authorino-6f48949686-sngj6\" (UID: \"96b9b7f3-d372-44ca-adfb-79546be9f85e\") " pod="kuadrant-system/authorino-6f48949686-sngj6"
Oct 29 22:51:06.904010 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:06.903974    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f48949686-sngj6"
Oct 29 22:51:07.214893 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:07.214860    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f48949686-sngj6"]
Oct 29 22:51:07.219120 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:51:07.219093    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96b9b7f3_d372_44ca_adfb_79546be9f85e.slice/crio-e5baa700601694102ce38c7d45d92b132a0dbf720b4f652fa6d08d638c7cb05e WatchSource:0}: Error finding container e5baa700601694102ce38c7d45d92b132a0dbf720b4f652fa6d08d638c7cb05e: Status 404 returned error can't find the container with id e5baa700601694102ce38c7d45d92b132a0dbf720b4f652fa6d08d638c7cb05e
Oct 29 22:51:07.245855 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:07.245827    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f48949686-sngj6" event={"ID":"96b9b7f3-d372-44ca-adfb-79546be9f85e","Type":"ContainerStarted","Data":"e5baa700601694102ce38c7d45d92b132a0dbf720b4f652fa6d08d638c7cb05e"}
Oct 29 22:51:07.511815 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:07.511724    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:07.514871 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:07.514845    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-fssgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 29 22:51:07.515013 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:07.514900    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc" podUID="42695ed6-95ae-457a-8822-8c9e303d06b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 29 22:51:07.641365 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:07.641341    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:08.254699 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:08.254652    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f48949686-sngj6" event={"ID":"96b9b7f3-d372-44ca-adfb-79546be9f85e","Type":"ContainerStarted","Data":"033102e1bbe2031f9dc4e426bbcfdd5e0998f1dbc86b8ddc66044440df6ce6fb"}
Oct 29 22:51:08.352621 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:08.352541    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f48949686-sngj6" podStartSLOduration=1.952470589 podStartE2EDuration="2.352519643s" totalImagesPullingTime="400.049054ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:08.303758437 +0000 UTC m=+471.646707184" watchObservedRunningTime="2026-10-29 22:51:08.352519643 +0000 UTC m=+471.695468385"
Oct 29 22:51:08.355425 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:08.355401    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-584dd45b57-w6sfl"]
Oct 29 22:51:08.355768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:08.355744    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-584dd45b57-w6sfl" podUID="8d860107-a315-4dde-852d-4b16c84dca10" containerName="authorino" containerID="cri-o://1f1c8d7252a0e25889c55a63f9eac50fe867957a2cdf534028c3e8626e67f61d" gracePeriod=30
Oct 29 22:51:08.515685 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:08.515609    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-fssgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 29 22:51:08.515853 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:08.515682    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc" podUID="42695ed6-95ae-457a-8822-8c9e303d06b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 29 22:51:08.651446 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:08.651404    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-584dd45b57-w6sfl"
Oct 29 22:51:08.694486 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:08.694449    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d860107-a315-4dde-852d-4b16c84dca10-kube-api-access-pbcd8\" (UniqueName: \"kubernetes.io/projected/8d860107-a315-4dde-852d-4b16c84dca10-kube-api-access-pbcd8\") pod \"8d860107-a315-4dde-852d-4b16c84dca10\" (UID: \"8d860107-a315-4dde-852d-4b16c84dca10\") "
Oct 29 22:51:08.694714 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:08.694597    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d860107-a315-4dde-852d-4b16c84dca10-tls-cert\" (UniqueName: \"kubernetes.io/secret/8d860107-a315-4dde-852d-4b16c84dca10-tls-cert\") pod \"8d860107-a315-4dde-852d-4b16c84dca10\" (UID: \"8d860107-a315-4dde-852d-4b16c84dca10\") "
Oct 29 22:51:08.698813 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:08.697040    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d860107-a315-4dde-852d-4b16c84dca10-kube-api-access-pbcd8" pod "8d860107-a315-4dde-852d-4b16c84dca10" (UID: "8d860107-a315-4dde-852d-4b16c84dca10"). InnerVolumeSpecName "kube-api-access-pbcd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:51:08.708358 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:08.708332    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d860107-a315-4dde-852d-4b16c84dca10-tls-cert" pod "8d860107-a315-4dde-852d-4b16c84dca10" (UID: "8d860107-a315-4dde-852d-4b16c84dca10"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:51:08.797822 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:08.795404    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbcd8\" (UniqueName: \"kubernetes.io/projected/8d860107-a315-4dde-852d-4b16c84dca10-kube-api-access-pbcd8\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:51:08.797822 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:08.795431    5428 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8d860107-a315-4dde-852d-4b16c84dca10-tls-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:51:09.046943 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:09.046859    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:09.220211 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:09.220170    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:09.285823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:09.285386    5428 generic.go:356] "Generic (PLEG): container finished" podID="8d860107-a315-4dde-852d-4b16c84dca10" containerID="1f1c8d7252a0e25889c55a63f9eac50fe867957a2cdf534028c3e8626e67f61d" exitCode=0
Oct 29 22:51:09.286021 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:09.285916    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-584dd45b57-w6sfl"
Oct 29 22:51:09.289646 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:09.286185    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-584dd45b57-w6sfl" event={"ID":"8d860107-a315-4dde-852d-4b16c84dca10","Type":"ContainerDied","Data":"1f1c8d7252a0e25889c55a63f9eac50fe867957a2cdf534028c3e8626e67f61d"}
Oct 29 22:51:09.289646 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:09.286242    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-584dd45b57-w6sfl" event={"ID":"8d860107-a315-4dde-852d-4b16c84dca10","Type":"ContainerDied","Data":"26c94076d0996344cb3c1870ee2f7d9d0e07a1f53588c40f071b061f2d149f7d"}
Oct 29 22:51:09.289646 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:09.286266    5428 scope.go:122] "RemoveContainer" containerID="1f1c8d7252a0e25889c55a63f9eac50fe867957a2cdf534028c3e8626e67f61d"
Oct 29 22:51:09.302260 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:09.302230    5428 scope.go:122] "RemoveContainer" containerID="1f1c8d7252a0e25889c55a63f9eac50fe867957a2cdf534028c3e8626e67f61d"
Oct 29 22:51:09.302832 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:51:09.302788    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f1c8d7252a0e25889c55a63f9eac50fe867957a2cdf534028c3e8626e67f61d\": container with ID starting with 1f1c8d7252a0e25889c55a63f9eac50fe867957a2cdf534028c3e8626e67f61d not found: ID does not exist" containerID="1f1c8d7252a0e25889c55a63f9eac50fe867957a2cdf534028c3e8626e67f61d"
Oct 29 22:51:09.302947 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:09.302840    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f1c8d7252a0e25889c55a63f9eac50fe867957a2cdf534028c3e8626e67f61d"} err="failed to get container status \"1f1c8d7252a0e25889c55a63f9eac50fe867957a2cdf534028c3e8626e67f61d\": rpc error: code = NotFound desc = could not find container \"1f1c8d7252a0e25889c55a63f9eac50fe867957a2cdf534028c3e8626e67f61d\": container with ID starting with 1f1c8d7252a0e25889c55a63f9eac50fe867957a2cdf534028c3e8626e67f61d not found: ID does not exist"
Oct 29 22:51:09.339768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:09.339711    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-584dd45b57-w6sfl"]
Oct 29 22:51:09.342903 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:09.342872    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-584dd45b57-w6sfl"]
Oct 29 22:51:09.515134 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:09.515101    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-fssgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 29 22:51:09.515309 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:09.515156    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc" podUID="42695ed6-95ae-457a-8822-8c9e303d06b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 29 22:51:10.515741 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:10.515702    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-fssgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 29 22:51:10.516117 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:10.515771    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc" podUID="42695ed6-95ae-457a-8822-8c9e303d06b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 29 22:51:10.775659 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:10.775579    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d860107-a315-4dde-852d-4b16c84dca10" path="/var/lib/kubelet/pods/8d860107-a315-4dde-852d-4b16c84dca10/volumes"
Oct 29 22:51:11.274646 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:11.274609    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:12.209140 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:12.209104    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:12.209867 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:12.209842    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:12.210788 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:12.210760    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:51:17.366903 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:51:17.366848    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:51:18.229408 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.229374    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"]
Oct 29 22:51:18.229840 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.229825    5428 cpu_manager.go:414] "containerMap: removing container" podUID="8d860107-a315-4dde-852d-4b16c84dca10" containerName="authorino"
Oct 29 22:51:18.229888 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.229841    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d860107-a315-4dde-852d-4b16c84dca10" containerName="authorino"
Oct 29 22:51:18.229934 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.229925    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d860107-a315-4dde-852d-4b16c84dca10" containerName="authorino"
Oct 29 22:51:18.234597 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.234578    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.262578 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.262551    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"]
Oct 29 22:51:18.295619 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.295583    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.295755 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.295625    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.295755 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.295652    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.295852 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.295767    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.295910 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.295888    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.295961 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.295946    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.296001 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.295987    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjz4z\" (UniqueName: \"kubernetes.io/projected/bf52a0e8-c739-47f8-9676-c5261e8d0f59-kube-api-access-rjz4z\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.296059 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.296041    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.296092 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.296073    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.397598 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.397567    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.398053 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.397606    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.398053 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.397654    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.398053 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.397677    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.398053 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.397699    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.398053 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.397735    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.398053 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.397823    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.398053 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.397849    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.398053 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.397895    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjz4z\" (UniqueName: \"kubernetes.io/projected/bf52a0e8-c739-47f8-9676-c5261e8d0f59-kube-api-access-rjz4z\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.398479 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.398261    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.398479 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.398325    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.398479 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.398387    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.398623 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.398508    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.398747 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.398730    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.400142 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.400116    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.400233 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.400217    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.419504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.419475    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.419716 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.419694    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjz4z\" (UniqueName: \"kubernetes.io/projected/bf52a0e8-c739-47f8-9676-c5261e8d0f59-kube-api-access-rjz4z\") pod \"maas-default-gateway-openshift-default-55fbd65575-98p7b\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.549839 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.549746    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:18.868727 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.868691    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"]
Oct 29 22:51:18.872579 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:51:18.872552    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf52a0e8_c739_47f8_9676_c5261e8d0f59.slice/crio-f91e539953b771366f040ed5d2a5ddc69becc1c4ea6653088eeadc4f32420785 WatchSource:0}: Error finding container f91e539953b771366f040ed5d2a5ddc69becc1c4ea6653088eeadc4f32420785: Status 404 returned error can't find the container with id f91e539953b771366f040ed5d2a5ddc69becc1c4ea6653088eeadc4f32420785
Oct 29 22:51:18.874766 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.874740    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:51:18.874848 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.874829    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:51:18.874896 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:18.874864    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:51:19.346069 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:19.345977    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b" event={"ID":"bf52a0e8-c739-47f8-9676-c5261e8d0f59","Type":"ContainerStarted","Data":"a60ccf3743cce0bf957b1afabc543f1a02f35a8ad2955c4b9bc4ed4b9b169e32"}
Oct 29 22:51:19.346069 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:19.346034    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b" event={"ID":"bf52a0e8-c739-47f8-9676-c5261e8d0f59","Type":"ContainerStarted","Data":"f91e539953b771366f040ed5d2a5ddc69becc1c4ea6653088eeadc4f32420785"}
Oct 29 22:51:19.403040 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:19.402985    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b" podStartSLOduration=1.402969438 podStartE2EDuration="1.402969438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:19.401340806 +0000 UTC m=+482.744289550" watchObservedRunningTime="2026-10-29 22:51:19.402969438 +0000 UTC m=+482.745918183"
Oct 29 22:51:19.549974 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:19.549939    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:19.551379 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:19.551351    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-98p7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:51:19.551501 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:19.551400    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b" podUID="bf52a0e8-c739-47f8-9676-c5261e8d0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 29 22:51:20.550528 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:20.550490    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-98p7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:51:20.550950 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:20.550545    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b" podUID="bf52a0e8-c739-47f8-9676-c5261e8d0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 29 22:51:21.353597 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:21.353561    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:21.482825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:21.481843    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:21.550882 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:21.550852    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-98p7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:51:21.551269 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:21.550910    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b" podUID="bf52a0e8-c739-47f8-9676-c5261e8d0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 29 22:51:22.550287 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:22.550251    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-98p7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:51:22.550466 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:22.550308    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b" podUID="bf52a0e8-c739-47f8-9676-c5261e8d0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 29 22:51:23.551129 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:23.551092    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-98p7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:51:23.551532 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:23.551157    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b" podUID="bf52a0e8-c739-47f8-9676-c5261e8d0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 29 22:51:23.598113 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:23.598079    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:24.550529 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:24.550494    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-98p7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:51:24.550710 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:24.550546    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b" podUID="bf52a0e8-c739-47f8-9676-c5261e8d0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 29 22:51:25.550899 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:25.550864    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-98p7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 29 22:51:25.551283 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:25.550922    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b" podUID="bf52a0e8-c739-47f8-9676-c5261e8d0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 29 22:51:27.114765 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:27.114733    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:27.115304 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:27.115257    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:27.115937 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:27.115917    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 22:51:33.199331 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.199299    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"]
Oct 29 22:51:33.202977 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.202959    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.226929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.226902    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"]
Oct 29 22:51:33.342655 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.342623    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.342655 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.342655    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.342886 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.342687    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.342886 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.342844    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3977ef-edc8-47f1-9779-c0f257e0e475-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.342955 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.342896    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkl9k\" (UniqueName: \"kubernetes.io/projected/9d3977ef-edc8-47f1-9779-c0f257e0e475-kube-api-access-mkl9k\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.342955 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.342940    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.343020 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.342993    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.343094 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.343037    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.343128 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.343105    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.444371 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.444336    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.444574 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.444393    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.444574 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.444416    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.444574 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.444461    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.444574 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.444507    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3977ef-edc8-47f1-9779-c0f257e0e475-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.444574 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.444556    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkl9k\" (UniqueName: \"kubernetes.io/projected/9d3977ef-edc8-47f1-9779-c0f257e0e475-kube-api-access-mkl9k\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.444880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.444587    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.444880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.444628    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.444880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.444685    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.445033 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.444908    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.445033 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.444966    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.445150 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.445126    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.445224 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.445178    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.445470 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.445447    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3977ef-edc8-47f1-9779-c0f257e0e475-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.446828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.446796    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.447255 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.447234    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.466262 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.466205    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.466426 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.466401    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkl9k\" (UniqueName: \"kubernetes.io/projected/9d3977ef-edc8-47f1-9779-c0f257e0e475-kube-api-access-mkl9k\") pod \"maas-default-gateway-openshift-default-55fbd65575-d9nr2\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.517854 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.517825    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:33.834740 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.834577    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"]
Oct 29 22:51:33.839383 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:51:33.838215    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3977ef_edc8_47f1_9779_c0f257e0e475.slice/crio-65204e703f0ce4b82c99e572eaca96df874b5f27cb9504a16ce13430cdc4781f WatchSource:0}: Error finding container 65204e703f0ce4b82c99e572eaca96df874b5f27cb9504a16ce13430cdc4781f: Status 404 returned error can't find the container with id 65204e703f0ce4b82c99e572eaca96df874b5f27cb9504a16ce13430cdc4781f
Oct 29 22:51:33.841726 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.841695    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:51:33.841831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.841759    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:51:33.841831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:33.841793    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:51:34.448059 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:34.448013    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2" event={"ID":"9d3977ef-edc8-47f1-9779-c0f257e0e475","Type":"ContainerStarted","Data":"9f4ce7755f4e6ac925fc9e59641164798bbfd8e42914af5a0e0ab59af2653b86"}
Oct 29 22:51:34.448432 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:34.448115    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2" event={"ID":"9d3977ef-edc8-47f1-9779-c0f257e0e475","Type":"ContainerStarted","Data":"65204e703f0ce4b82c99e572eaca96df874b5f27cb9504a16ce13430cdc4781f"}
Oct 29 22:51:34.480511 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:34.480476    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:34.508458 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:34.508375    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2" podStartSLOduration=1.508360427 podStartE2EDuration="1.508360427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:34.506490967 +0000 UTC m=+497.849439712" watchObservedRunningTime="2026-10-29 22:51:34.508360427 +0000 UTC m=+497.851309173"
Oct 29 22:51:34.518824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:34.518780    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:34.520059 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:34.520031    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-d9nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:51:34.520179 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:34.520080    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2" podUID="9d3977ef-edc8-47f1-9779-c0f257e0e475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 29 22:51:34.625431 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:34.625382    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:34.749372 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:34.749296    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:34.988196 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:34.988167    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:35.228524 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.228482    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:35.309607 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.309573    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"]
Oct 29 22:51:35.313531 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.313501    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.330095 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.330062    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-865d946f-openshift-default-dockercfg-kptlz\""
Oct 29 22:51:35.341815 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.341780    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"]
Oct 29 22:51:35.392837 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.391209    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:35.491759 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.491665    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-255tx\" (UniqueName: \"kubernetes.io/projected/fd186355-c594-4b2b-b716-21ced7841757-kube-api-access-255tx\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.492410 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.492371    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-istio-envoy\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.492520 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.492422    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd186355-c594-4b2b-b716-21ced7841757-istio-token\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.492561 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.492521    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-istio-data\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.492692 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.492670    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-workload-certs\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.492765 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.492731    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd186355-c594-4b2b-b716-21ced7841757-istiod-ca-cert\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.492859 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.492842    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-workload-socket\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.492929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.492908    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-credential-socket\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.492997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.492981    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd186355-c594-4b2b-b716-21ced7841757-istio-podinfo\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.518769 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.518739    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-d9nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:51:35.518943 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.518787    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2" podUID="9d3977ef-edc8-47f1-9779-c0f257e0e475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 29 22:51:35.524912 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.524878    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:35.594582 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.594543    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-workload-certs\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.594582 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.594585    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd186355-c594-4b2b-b716-21ced7841757-istiod-ca-cert\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.594906 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.594639    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-workload-socket\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.594906 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.594673    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-credential-socket\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.594906 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.594742    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd186355-c594-4b2b-b716-21ced7841757-istio-podinfo\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.595094 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.595045    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-255tx\" (UniqueName: \"kubernetes.io/projected/fd186355-c594-4b2b-b716-21ced7841757-kube-api-access-255tx\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.595148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.595093    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-istio-envoy\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.595148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.595117    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd186355-c594-4b2b-b716-21ced7841757-istio-token\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.595248 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.595166    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-istio-data\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.595248 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.595197    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-workload-certs\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.595348 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.595276    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-credential-socket\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.595722 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.595476    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-istio-data\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.595722 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.595679    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd186355-c594-4b2b-b716-21ced7841757-istiod-ca-cert\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.595894 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.595793    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-workload-socket\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.597458 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.597434    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd186355-c594-4b2b-b716-21ced7841757-istio-podinfo\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.597618 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.597603    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-istio-envoy\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.611700 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.611673    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"]
Oct 29 22:51:35.616012 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.615993    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.623645 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.623622    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd186355-c594-4b2b-b716-21ced7841757-istio-token\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.624991 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.624971    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-255tx\" (UniqueName: \"kubernetes.io/projected/fd186355-c594-4b2b-b716-21ced7841757-kube-api-access-255tx\") pod \"e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") " pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.629062 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.629037    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:35.644294 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.644261    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"]
Oct 29 22:51:35.677238 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.677185    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:35.716302 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.716264    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e406071d-openshift-default-dockercfg-8lvl7\""
Oct 29 22:51:35.802774 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.802658    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-workload-certs\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.802774 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.802742    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-podinfo\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.803053 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.802905    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-credential-socket\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.803053 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.802947    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-envoy\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.803053 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.802992    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-workload-socket\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.803221 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.803057    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwdqt\" (UniqueName: \"kubernetes.io/projected/887f4003-a75f-4799-8c46-2460a9f89f8e-kube-api-access-kwdqt\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.803221 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.803094    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-data\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.803221 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.803143    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-token\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.803388 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.803216    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887f4003-a75f-4799-8c46-2460a9f89f8e-istiod-ca-cert\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.804851 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.804823    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:35.904016 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.903984    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwdqt\" (UniqueName: \"kubernetes.io/projected/887f4003-a75f-4799-8c46-2460a9f89f8e-kube-api-access-kwdqt\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.904199 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.904026    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-data\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.904199 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.904052    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-token\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.904313 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.904250    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887f4003-a75f-4799-8c46-2460a9f89f8e-istiod-ca-cert\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.904372 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.904309    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-workload-certs\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.904372 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.904357    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-podinfo\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.904487 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.904451    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-data\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.904487 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.904470    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-credential-socket\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.904596 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.904546    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-envoy\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.904596 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.904577    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-workload-socket\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.905049 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.904903    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-workload-certs\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.905049 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.904933    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-credential-socket\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.905049 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.905019    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-workload-socket\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.906286 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.906266    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887f4003-a75f-4799-8c46-2460a9f89f8e-istiod-ca-cert\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.907871 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.907851    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-envoy\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.909372 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.909352    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-podinfo\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.926797 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.926763    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwdqt\" (UniqueName: \"kubernetes.io/projected/887f4003-a75f-4799-8c46-2460a9f89f8e-kube-api-access-kwdqt\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.926966 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.926946    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-token\") pod \"e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") " pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.944553 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.944531    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:35.984058 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.984022    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:35.990401 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.990316    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"]
Oct 29 22:51:35.994429 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:51:35.994403    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd186355_c594_4b2b_b716_21ced7841757.slice/crio-2af4bda0aa4ef654ea16a97cc6cfa3872db4c1646fb6391826bbb873baee65d7 WatchSource:0}: Error finding container 2af4bda0aa4ef654ea16a97cc6cfa3872db4c1646fb6391826bbb873baee65d7: Status 404 returned error can't find the container with id 2af4bda0aa4ef654ea16a97cc6cfa3872db4c1646fb6391826bbb873baee65d7
Oct 29 22:51:35.997131 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.997098    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:51:35.997220 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.997182    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:51:35.997260 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:35.997217    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:51:36.196821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:36.196252    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:36.327670 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:36.327466    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:36.328596 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:36.328566    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"]
Oct 29 22:51:36.337253 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:36.337206    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:51:36.337508 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:36.337482    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:51:36.337623 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:36.337552    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:51:36.494463 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:36.494276    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs" event={"ID":"887f4003-a75f-4799-8c46-2460a9f89f8e","Type":"ContainerStarted","Data":"c3d0d38ef97f1c8e80dae32b60b12317e232b9dbc412c47fad567d55f5e72c55"}
Oct 29 22:51:36.494463 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:36.494412    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs" event={"ID":"887f4003-a75f-4799-8c46-2460a9f89f8e","Type":"ContainerStarted","Data":"cf807496f7debd786a7a2292c0a75cec96f133f0de9f9c9fde4cd1d65d3a130e"}
Oct 29 22:51:36.499782 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:36.499750    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn" event={"ID":"fd186355-c594-4b2b-b716-21ced7841757","Type":"ContainerStarted","Data":"d0cb063f35742e4738e0e3fb88629419b88866f7719cdf487f10ec82e54f5e2d"}
Oct 29 22:51:36.500044 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:36.500024    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn" event={"ID":"fd186355-c594-4b2b-b716-21ced7841757","Type":"ContainerStarted","Data":"2af4bda0aa4ef654ea16a97cc6cfa3872db4c1646fb6391826bbb873baee65d7"}
Oct 29 22:51:36.518231 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:36.518202    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-d9nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:51:36.518384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:36.518260    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2" podUID="9d3977ef-edc8-47f1-9779-c0f257e0e475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 29 22:51:36.538180 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:36.538134    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs" podStartSLOduration=1.538121008 podStartE2EDuration="1.538121008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:36.535490102 +0000 UTC m=+499.878438846" watchObservedRunningTime="2026-10-29 22:51:36.538121008 +0000 UTC m=+499.881069754"
Oct 29 22:51:36.590973 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:36.590564    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn" podStartSLOduration=1.590542261 podStartE2EDuration="1.590542261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:36.581310146 +0000 UTC m=+499.924258906" watchObservedRunningTime="2026-10-29 22:51:36.590542261 +0000 UTC m=+499.933491007"
Oct 29 22:51:36.629410 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:36.629381    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:36.747642 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:36.747604    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:36.985029 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:36.984982    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:37.384811 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:37.384769    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:37.519134 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:37.519100    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-d9nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:51:37.519591 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:37.519153    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2" podUID="9d3977ef-edc8-47f1-9779-c0f257e0e475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 29 22:51:37.571705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:37.571666    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:37.634687 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:37.634658    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:37.828844 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:37.828746    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:37.976029 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:37.975995    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:37.990447 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:37.990422    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:38.206649 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:38.206607    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:38.342245 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:38.342210    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:38.484715 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:38.484617    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:38.513151 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:38.513127    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:38.513334 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:38.513180    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:38.513980 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:38.513955    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:51:38.514099 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:38.514019    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:51:38.518347 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:38.518319    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-d9nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:51:38.518427 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:38.518360    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2" podUID="9d3977ef-edc8-47f1-9779-c0f257e0e475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 29 22:51:38.621977 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:38.621945    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:39.398141 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.398098    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:39.518252 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.518217    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-d9nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:51:39.518438 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.518274    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2" podUID="9d3977ef-edc8-47f1-9779-c0f257e0e475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 29 22:51:39.554111 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.554075    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:39.705847 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.705758    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st"]
Oct 29 22:51:39.711472 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.711435    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st"
Oct 29 22:51:39.722028 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.722002    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:39.732628 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.732384    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st"]
Oct 29 22:51:39.741911 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.741889    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5db969e0-dockercfg-tfmpr\""
Oct 29 22:51:39.742037 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.742017    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5db969e0\""
Oct 29 22:51:39.750446 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.750411    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsg4n\" (UniqueName: \"kubernetes.io/projected/e4dfd4c3-c1a6-4438-a299-d006b1107e35-kube-api-access-nsg4n\") pod \"payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st\" (UID: \"e4dfd4c3-c1a6-4438-a299-d006b1107e35\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st"
Oct 29 22:51:39.750551 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.750535    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4dfd4c3-c1a6-4438-a299-d006b1107e35-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st\" (UID: \"e4dfd4c3-c1a6-4438-a299-d006b1107e35\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st"
Oct 29 22:51:39.840373 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.840335    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8"]
Oct 29 22:51:39.844951 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.844929    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8"
Oct 29 22:51:39.851220 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.851191    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f706efa1-73c8-4a34-a0b4-7be82c2be566-plugins-config-volume\") pod \"payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8\" (UID: \"f706efa1-73c8-4a34-a0b4-7be82c2be566\") " pod="openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8"
Oct 29 22:51:39.851348 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.851261    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsg4n\" (UniqueName: \"kubernetes.io/projected/e4dfd4c3-c1a6-4438-a299-d006b1107e35-kube-api-access-nsg4n\") pod \"payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st\" (UID: \"e4dfd4c3-c1a6-4438-a299-d006b1107e35\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st"
Oct 29 22:51:39.851348 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.851326    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bwxt\" (UniqueName: \"kubernetes.io/projected/f706efa1-73c8-4a34-a0b4-7be82c2be566-kube-api-access-4bwxt\") pod \"payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8\" (UID: \"f706efa1-73c8-4a34-a0b4-7be82c2be566\") " pod="openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8"
Oct 29 22:51:39.851464 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.851366    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4dfd4c3-c1a6-4438-a299-d006b1107e35-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st\" (UID: \"e4dfd4c3-c1a6-4438-a299-d006b1107e35\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st"
Oct 29 22:51:39.852165 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.852140    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4dfd4c3-c1a6-4438-a299-d006b1107e35-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st\" (UID: \"e4dfd4c3-c1a6-4438-a299-d006b1107e35\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st"
Oct 29 22:51:39.880117 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.880086    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsg4n\" (UniqueName: \"kubernetes.io/projected/e4dfd4c3-c1a6-4438-a299-d006b1107e35-kube-api-access-nsg4n\") pod \"payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st\" (UID: \"e4dfd4c3-c1a6-4438-a299-d006b1107e35\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st"
Oct 29 22:51:39.901734 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.901708    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8"]
Oct 29 22:51:39.923689 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.923655    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:39.952071 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.952043    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f706efa1-73c8-4a34-a0b4-7be82c2be566-plugins-config-volume\") pod \"payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8\" (UID: \"f706efa1-73c8-4a34-a0b4-7be82c2be566\") " pod="openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8"
Oct 29 22:51:39.952225 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.952118    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bwxt\" (UniqueName: \"kubernetes.io/projected/f706efa1-73c8-4a34-a0b4-7be82c2be566-kube-api-access-4bwxt\") pod \"payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8\" (UID: \"f706efa1-73c8-4a34-a0b4-7be82c2be566\") " pod="openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8"
Oct 29 22:51:39.952785 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.952766    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f706efa1-73c8-4a34-a0b4-7be82c2be566-plugins-config-volume\") pod \"payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8\" (UID: \"f706efa1-73c8-4a34-a0b4-7be82c2be566\") " pod="openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8"
Oct 29 22:51:39.977992 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:39.977907    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bwxt\" (UniqueName: \"kubernetes.io/projected/f706efa1-73c8-4a34-a0b4-7be82c2be566-kube-api-access-4bwxt\") pod \"payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8\" (UID: \"f706efa1-73c8-4a34-a0b4-7be82c2be566\") " pod="openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8"
Oct 29 22:51:40.027118 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:40.027091    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st"
Oct 29 22:51:40.158090 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:40.158052    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8"
Oct 29 22:51:40.340568 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:40.340539    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st"]
Oct 29 22:51:40.342515 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:51:40.342479    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4dfd4c3_c1a6_4438_a299_d006b1107e35.slice/crio-e967d7cf5bc47babafb2bc5079f94c0f3b2251f32106b7483b021d03d107f2dc WatchSource:0}: Error finding container e967d7cf5bc47babafb2bc5079f94c0f3b2251f32106b7483b021d03d107f2dc: Status 404 returned error can't find the container with id e967d7cf5bc47babafb2bc5079f94c0f3b2251f32106b7483b021d03d107f2dc
Oct 29 22:51:40.459690 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:40.459657    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8"]
Oct 29 22:51:40.466761 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:51:40.463921    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf706efa1_73c8_4a34_a0b4_7be82c2be566.slice/crio-1e296b9a4f2eec5adc08a98451f922d4310c7669194f2a4e104352c74b7ba6c2 WatchSource:0}: Error finding container 1e296b9a4f2eec5adc08a98451f922d4310c7669194f2a4e104352c74b7ba6c2: Status 404 returned error can't find the container with id 1e296b9a4f2eec5adc08a98451f922d4310c7669194f2a4e104352c74b7ba6c2
Oct 29 22:51:40.518746 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:40.518722    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-d9nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:51:40.518879 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:40.518766    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2" podUID="9d3977ef-edc8-47f1-9779-c0f257e0e475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 29 22:51:40.525952 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:40.525919    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st" event={"ID":"e4dfd4c3-c1a6-4438-a299-d006b1107e35","Type":"ContainerStarted","Data":"e23544037321ac7bec6e627178c437327a8447da350ded05a9ed06c6fe8a8330"}
Oct 29 22:51:40.526063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:40.525976    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st" event={"ID":"e4dfd4c3-c1a6-4438-a299-d006b1107e35","Type":"ContainerStarted","Data":"e967d7cf5bc47babafb2bc5079f94c0f3b2251f32106b7483b021d03d107f2dc"}
Oct 29 22:51:40.526063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:40.526008    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st"
Oct 29 22:51:40.527500 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:40.527475    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8" event={"ID":"f706efa1-73c8-4a34-a0b4-7be82c2be566","Type":"ContainerStarted","Data":"1e296b9a4f2eec5adc08a98451f922d4310c7669194f2a4e104352c74b7ba6c2"}
Oct 29 22:51:40.574770 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:40.574710    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st" podStartSLOduration=1.574692512 podStartE2EDuration="1.574692512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:40.571295651 +0000 UTC m=+503.914244397" watchObservedRunningTime="2026-10-29 22:51:40.574692512 +0000 UTC m=+503.917641261"
Oct 29 22:51:40.790399 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:40.790363    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:40.908273 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:40.908214    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:41.286867 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.286823    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:41.399571 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.399527    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:41.521440 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.521407    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp"]
Oct 29 22:51:41.533393 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.533196    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp"
Oct 29 22:51:41.539646 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.539573    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp"]
Oct 29 22:51:41.552849 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.542229    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8" event={"ID":"f706efa1-73c8-4a34-a0b4-7be82c2be566","Type":"ContainerStarted","Data":"c22a5370913c369b0252c9687414da250213777a5dc5845f1bf50706a3f355c8"}
Oct 29 22:51:41.554230 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.554208    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8"
Oct 29 22:51:41.567077 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.565958    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-e406071d\""
Oct 29 22:51:41.569462 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.569355    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:41.589644 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.589604    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3f2e47-48c7-4691-938d-5424e634f2f5-plugins-config-volume\") pod \"payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp\" (UID: \"6d3f2e47-48c7-4691-938d-5424e634f2f5\") " pod="openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp"
Oct 29 22:51:41.589829 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.589783    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvnz7\" (UniqueName: \"kubernetes.io/projected/6d3f2e47-48c7-4691-938d-5424e634f2f5-kube-api-access-nvnz7\") pod \"payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp\" (UID: \"6d3f2e47-48c7-4691-938d-5424e634f2f5\") " pod="openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp"
Oct 29 22:51:41.615847 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.615779    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8" podStartSLOduration=2.615767127 podStartE2EDuration="2.615767127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:41.613434051 +0000 UTC m=+504.956382798" watchObservedRunningTime="2026-10-29 22:51:41.615767127 +0000 UTC m=+504.958715871"
Oct 29 22:51:41.639522 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.639495    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-e406071d-dockercfg-p4t4q\""
Oct 29 22:51:41.690769 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.690731    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvnz7\" (UniqueName: \"kubernetes.io/projected/6d3f2e47-48c7-4691-938d-5424e634f2f5-kube-api-access-nvnz7\") pod \"payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp\" (UID: \"6d3f2e47-48c7-4691-938d-5424e634f2f5\") " pod="openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp"
Oct 29 22:51:41.690945 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.690871    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3f2e47-48c7-4691-938d-5424e634f2f5-plugins-config-volume\") pod \"payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp\" (UID: \"6d3f2e47-48c7-4691-938d-5424e634f2f5\") " pod="openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp"
Oct 29 22:51:41.691544 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.691517    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3f2e47-48c7-4691-938d-5424e634f2f5-plugins-config-volume\") pod \"payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp\" (UID: \"6d3f2e47-48c7-4691-938d-5424e634f2f5\") " pod="openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp"
Oct 29 22:51:41.693291 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.693177    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:41.712495 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.712469    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvnz7\" (UniqueName: \"kubernetes.io/projected/6d3f2e47-48c7-4691-938d-5424e634f2f5-kube-api-access-nvnz7\") pod \"payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp\" (UID: \"6d3f2e47-48c7-4691-938d-5424e634f2f5\") " pod="openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp"
Oct 29 22:51:41.805703 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.805610    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:41.878438 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:41.878395    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp"
Oct 29 22:51:42.191493 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.191460    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp"]
Oct 29 22:51:42.210221 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:51:42.210183    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3f2e47_48c7_4691_938d_5424e634f2f5.slice/crio-50bd23e7036ef9a4a3cc81cf5f6b0f0252bea1eb80e04a12153a76e376f6cf5e WatchSource:0}: Error finding container 50bd23e7036ef9a4a3cc81cf5f6b0f0252bea1eb80e04a12153a76e376f6cf5e: Status 404 returned error can't find the container with id 50bd23e7036ef9a4a3cc81cf5f6b0f0252bea1eb80e04a12153a76e376f6cf5e
Oct 29 22:51:42.263672 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.263642    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:42.314451 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.314419    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:42.314870 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.314848    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:42.315556 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.315534    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 22:51:42.374906 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.374871    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg"]
Oct 29 22:51:42.378655 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.378633    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg"
Oct 29 22:51:42.391100 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.391076    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:42.394068 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.394051    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-865d946f-dockercfg-m2s6t\""
Oct 29 22:51:42.400217 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.400187    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg"]
Oct 29 22:51:42.424365 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.424339    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-865d946f\""
Oct 29 22:51:42.509097 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.509012    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r98n4\" (UniqueName: \"kubernetes.io/projected/47d1fe46-10af-40db-9f1c-9c696185d3ea-kube-api-access-r98n4\") pod \"payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg\" (UID: \"47d1fe46-10af-40db-9f1c-9c696185d3ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg"
Oct 29 22:51:42.509097 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.509082    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d1fe46-10af-40db-9f1c-9c696185d3ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg\" (UID: \"47d1fe46-10af-40db-9f1c-9c696185d3ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg"
Oct 29 22:51:42.551170 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.551129    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp" event={"ID":"6d3f2e47-48c7-4691-938d-5424e634f2f5","Type":"ContainerStarted","Data":"d82ee008b9bf72c12bac4b4b76a0db899d583d785ac5f7d54fb434326d496247"}
Oct 29 22:51:42.551414 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.551221    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp" event={"ID":"6d3f2e47-48c7-4691-938d-5424e634f2f5","Type":"ContainerStarted","Data":"50bd23e7036ef9a4a3cc81cf5f6b0f0252bea1eb80e04a12153a76e376f6cf5e"}
Oct 29 22:51:42.551674 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.551644    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp"
Oct 29 22:51:42.589742 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.589681    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp" podStartSLOduration=1.5896650540000001 podStartE2EDuration="1.589665054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:42.5874569 +0000 UTC m=+505.930405644" watchObservedRunningTime="2026-10-29 22:51:42.589665054 +0000 UTC m=+505.932613815"
Oct 29 22:51:42.610759 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.610716    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r98n4\" (UniqueName: \"kubernetes.io/projected/47d1fe46-10af-40db-9f1c-9c696185d3ea-kube-api-access-r98n4\") pod \"payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg\" (UID: \"47d1fe46-10af-40db-9f1c-9c696185d3ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg"
Oct 29 22:51:42.611081 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.611053    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d1fe46-10af-40db-9f1c-9c696185d3ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg\" (UID: \"47d1fe46-10af-40db-9f1c-9c696185d3ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg"
Oct 29 22:51:42.611685 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.611658    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d1fe46-10af-40db-9f1c-9c696185d3ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg\" (UID: \"47d1fe46-10af-40db-9f1c-9c696185d3ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg"
Oct 29 22:51:42.637106 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.637076    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r98n4\" (UniqueName: \"kubernetes.io/projected/47d1fe46-10af-40db-9f1c-9c696185d3ea-kube-api-access-r98n4\") pod \"payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg\" (UID: \"47d1fe46-10af-40db-9f1c-9c696185d3ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg"
Oct 29 22:51:42.692042 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.692004    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg"
Oct 29 22:51:42.851358 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:42.851322    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:43.078218 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:43.067055    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg"]
Oct 29 22:51:43.086269 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:43.081754    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:43.210790 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:43.210760    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:43.352305 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:43.352216    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:43.559426 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:43.559379    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg" event={"ID":"47d1fe46-10af-40db-9f1c-9c696185d3ea","Type":"ContainerStarted","Data":"ed92fea288cf319b805c850befda3128fb3cd94958509d11f4e3183c73d19fd3"}
Oct 29 22:51:43.559604 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:43.559448    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg" event={"ID":"47d1fe46-10af-40db-9f1c-9c696185d3ea","Type":"ContainerStarted","Data":"80940b1633ea1d76d6750b2623b26dede89be99e7d3ee775273b31f87c226903"}
Oct 29 22:51:43.559604 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:43.559484    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg"
Oct 29 22:51:43.561349 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:43.561320    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:43.598203 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:43.598146    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg" podStartSLOduration=1.598131953 podStartE2EDuration="1.598131953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:43.597085682 +0000 UTC m=+506.940034444" watchObservedRunningTime="2026-10-29 22:51:43.598131953 +0000 UTC m=+506.941080697"
Oct 29 22:51:43.672401 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:43.672372    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:43.795941 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:43.795877    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:44.029509 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.029419    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:44.143250 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.143216    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:44.466055 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.466022    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"]
Oct 29 22:51:44.470725 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.470692    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.503675 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.503641    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"]
Oct 29 22:51:44.510706 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.510681    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-99dfb0-openshift-default-dockercfg-72qwh\""
Oct 29 22:51:44.535375 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.535346    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:44.538650 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.538624    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.538772 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.538658    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.538772 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.538678    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.538907 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.538832    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.538962 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.538923    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.539027 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.538970    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.539087 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.539022    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7zgm\" (UniqueName: \"kubernetes.io/projected/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-kube-api-access-c7zgm\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.539087 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.539058    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.539187 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.539088    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.639758 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.639721    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.639963 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.639840    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.639963 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.639888    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.639963 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.639931    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7zgm\" (UniqueName: \"kubernetes.io/projected/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-kube-api-access-c7zgm\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.640170 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.639995    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.640170 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.640058    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.640170 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.640103    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.640170 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.640151    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.640358 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.640179    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.640358 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.640240    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.640781 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.640514    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.640781 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.640574    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.640969 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.640905    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.641266 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.641243    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.642793 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.642764    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.643184 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.643164    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.661334 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.661312    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.661417 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.661398    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7zgm\" (UniqueName: \"kubernetes.io/projected/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-kube-api-access-c7zgm\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.785912 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.785836    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:44.999574 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:44.999542    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:45.108815 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:45.108778    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"]
Oct 29 22:51:45.110853 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:51:45.110826    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c524b88_93cc_4890_aca0_d1dbd4b2f2dc.slice/crio-6be51cbb85875e2ab45ab83be7cf881d7ea23e5d8e355312fcfc2db95ff48836 WatchSource:0}: Error finding container 6be51cbb85875e2ab45ab83be7cf881d7ea23e5d8e355312fcfc2db95ff48836: Status 404 returned error can't find the container with id 6be51cbb85875e2ab45ab83be7cf881d7ea23e5d8e355312fcfc2db95ff48836
Oct 29 22:51:45.113234 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:45.113204    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:51:45.113311 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:45.113270    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:51:45.113311 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:45.113301    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:51:45.572987 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:45.572953    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp" event={"ID":"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc","Type":"ContainerStarted","Data":"122b37eccf0ed7fcb6d783ae4cb929f99107edc0f7a1a2e5babe95c59fd99637"}
Oct 29 22:51:45.573181 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:45.573015    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp" event={"ID":"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc","Type":"ContainerStarted","Data":"6be51cbb85875e2ab45ab83be7cf881d7ea23e5d8e355312fcfc2db95ff48836"}
Oct 29 22:51:45.636392 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:45.636342    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp" podStartSLOduration=1.636328771 podStartE2EDuration="1.636328771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:45.624014056 +0000 UTC m=+508.966962801" watchObservedRunningTime="2026-10-29 22:51:45.636328771 +0000 UTC m=+508.979277516"
Oct 29 22:51:45.786372 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:45.786331    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:45.952471 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:45.952439    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:46.151158 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:46.151110    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:46.290270 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:46.290186    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:46.323764 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:46.323736    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r"]
Oct 29 22:51:46.327891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:46.327875    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r"
Oct 29 22:51:46.347254 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:46.347221    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8042816e-dockercfg-qxwdq\""
Oct 29 22:51:46.349709 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:46.349671    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r"]
Oct 29 22:51:46.402125 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:46.373389    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8042816e\""
Oct 29 22:51:46.461388 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:46.461348    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqlvz\" (UniqueName: \"kubernetes.io/projected/909a27ab-47f9-4814-b653-b6c6959ad616-kube-api-access-kqlvz\") pod \"payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r\" (UID: \"909a27ab-47f9-4814-b653-b6c6959ad616\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r"
Oct 29 22:51:46.461552 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:46.461401    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909a27ab-47f9-4814-b653-b6c6959ad616-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r\" (UID: \"909a27ab-47f9-4814-b653-b6c6959ad616\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r"
Oct 29 22:51:46.568873 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:46.568460    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqlvz\" (UniqueName: \"kubernetes.io/projected/909a27ab-47f9-4814-b653-b6c6959ad616-kube-api-access-kqlvz\") pod \"payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r\" (UID: \"909a27ab-47f9-4814-b653-b6c6959ad616\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r"
Oct 29 22:51:46.568873 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:46.568534    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909a27ab-47f9-4814-b653-b6c6959ad616-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r\" (UID: \"909a27ab-47f9-4814-b653-b6c6959ad616\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r"
Oct 29 22:51:46.577710 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:46.569442    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909a27ab-47f9-4814-b653-b6c6959ad616-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r\" (UID: \"909a27ab-47f9-4814-b653-b6c6959ad616\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r"
Oct 29 22:51:46.591357 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:46.591323    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqlvz\" (UniqueName: \"kubernetes.io/projected/909a27ab-47f9-4814-b653-b6c6959ad616-kube-api-access-kqlvz\") pod \"payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r\" (UID: \"909a27ab-47f9-4814-b653-b6c6959ad616\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r"
Oct 29 22:51:46.642269 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:46.642241    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r"
Oct 29 22:51:46.793279 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:46.793251    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:46.942907 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:46.942871    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r"]
Oct 29 22:51:46.946016 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:51:46.945990    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909a27ab_47f9_4814_b653_b6c6959ad616.slice/crio-545ad220747248d65bbc3cbce7b88f59a5df61b34e25a39d5ceb53a196dfdee2 WatchSource:0}: Error finding container 545ad220747248d65bbc3cbce7b88f59a5df61b34e25a39d5ceb53a196dfdee2: Status 404 returned error can't find the container with id 545ad220747248d65bbc3cbce7b88f59a5df61b34e25a39d5ceb53a196dfdee2
Oct 29 22:51:47.365937 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:51:47.365897    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:51:47.591470 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:47.591437    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r" event={"ID":"909a27ab-47f9-4814-b653-b6c6959ad616","Type":"ContainerStarted","Data":"823057e80cf90960a151d25d86047badcc83fb3c410e24825c0aa7b8ccd2c9fa"}
Oct 29 22:51:47.591648 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:47.591493    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r" event={"ID":"909a27ab-47f9-4814-b653-b6c6959ad616","Type":"ContainerStarted","Data":"545ad220747248d65bbc3cbce7b88f59a5df61b34e25a39d5ceb53a196dfdee2"}
Oct 29 22:51:47.591648 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:47.591631    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r"
Oct 29 22:51:47.591919 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:47.591903    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:47.592899 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:47.592879    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:51:47.631576 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:47.631497    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:47.637300 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:47.637246    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r" podStartSLOduration=1.637232338 podStartE2EDuration="1.637232338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:47.633987089 +0000 UTC m=+510.976935835" watchObservedRunningTime="2026-10-29 22:51:47.637232338 +0000 UTC m=+510.980181083"
Oct 29 22:51:47.747456 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:47.747419    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:47.793654 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:47.793617    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs"]
Oct 29 22:51:47.797609 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:47.797585    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs"
Oct 29 22:51:47.811289 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:47.811262    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-e57216\""
Oct 29 22:51:47.811406 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:47.811284    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-e57216-dockercfg-ctfbl\""
Oct 29 22:51:47.818360 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:47.818332    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs"]
Oct 29 22:51:47.884601 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:47.884505    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4lvn\" (UniqueName: \"kubernetes.io/projected/f083378a-6a46-4573-9c83-ec5af5a6e153-kube-api-access-k4lvn\") pod \"payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs\" (UID: \"f083378a-6a46-4573-9c83-ec5af5a6e153\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs"
Oct 29 22:51:47.884949 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:47.884919    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f083378a-6a46-4573-9c83-ec5af5a6e153-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs\" (UID: \"f083378a-6a46-4573-9c83-ec5af5a6e153\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs"
Oct 29 22:51:47.986161 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:47.986127    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4lvn\" (UniqueName: \"kubernetes.io/projected/f083378a-6a46-4573-9c83-ec5af5a6e153-kube-api-access-k4lvn\") pod \"payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs\" (UID: \"f083378a-6a46-4573-9c83-ec5af5a6e153\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs"
Oct 29 22:51:47.986313 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:47.986193    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f083378a-6a46-4573-9c83-ec5af5a6e153-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs\" (UID: \"f083378a-6a46-4573-9c83-ec5af5a6e153\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs"
Oct 29 22:51:47.986876 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:47.986858    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f083378a-6a46-4573-9c83-ec5af5a6e153-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs\" (UID: \"f083378a-6a46-4573-9c83-ec5af5a6e153\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs"
Oct 29 22:51:48.008789 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:48.008762    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4lvn\" (UniqueName: \"kubernetes.io/projected/f083378a-6a46-4573-9c83-ec5af5a6e153-kube-api-access-k4lvn\") pod \"payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs\" (UID: \"f083378a-6a46-4573-9c83-ec5af5a6e153\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs"
Oct 29 22:51:48.112492 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:48.112466    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs"
Oct 29 22:51:48.132018 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:48.131986    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:48.268976 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:48.268913    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:48.419945 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:48.417657    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:48.419945 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:48.419500    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs"]
Oct 29 22:51:48.427847 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:51:48.427780    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf083378a_6a46_4573_9c83_ec5af5a6e153.slice/crio-3e14c19e745486edf682e885112bbd58e588d4bbaa34b15fef22c86cf189d4dd WatchSource:0}: Error finding container 3e14c19e745486edf682e885112bbd58e588d4bbaa34b15fef22c86cf189d4dd: Status 404 returned error can't find the container with id 3e14c19e745486edf682e885112bbd58e588d4bbaa34b15fef22c86cf189d4dd
Oct 29 22:51:48.606502 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:48.606458    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs" event={"ID":"f083378a-6a46-4573-9c83-ec5af5a6e153","Type":"ContainerStarted","Data":"46235af407a54886a4fe4cbc279ecae379a055eb4b29f8a906e8324905a6483e"}
Oct 29 22:51:48.606689 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:48.606554    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs" event={"ID":"f083378a-6a46-4573-9c83-ec5af5a6e153","Type":"ContainerStarted","Data":"3e14c19e745486edf682e885112bbd58e588d4bbaa34b15fef22c86cf189d4dd"}
Oct 29 22:51:48.606738 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:48.606690    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs"
Oct 29 22:51:48.647529 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:48.647471    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs" podStartSLOduration=1.6474538490000001 podStartE2EDuration="1.647453849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:48.645523273 +0000 UTC m=+511.988472018" watchObservedRunningTime="2026-10-29 22:51:48.647453849 +0000 UTC m=+511.990402594"
Oct 29 22:51:50.707929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:50.707894    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:51.442660 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:51.442625    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:51.544614 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:51.544588    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st"
Oct 29 22:51:51.587635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:51.587593    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:51.618751 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:51.618719    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf"]
Oct 29 22:51:51.623287 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:51.623266    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf"
Oct 29 22:51:51.643342 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:51.643310    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf"]
Oct 29 22:51:51.689882 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:51.683265    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-99dfb0\""
Oct 29 22:51:51.689882 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:51.683406    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-99dfb0-dockercfg-bsszb\""
Oct 29 22:51:51.733477 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:51.732377    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2032f74c-64e7-4c12-a969-c5b7eb33eb53-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf\" (UID: \"2032f74c-64e7-4c12-a969-c5b7eb33eb53\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf"
Oct 29 22:51:51.733477 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:51.732725    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr7ds\" (UniqueName: \"kubernetes.io/projected/2032f74c-64e7-4c12-a969-c5b7eb33eb53-kube-api-access-zr7ds\") pod \"payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf\" (UID: \"2032f74c-64e7-4c12-a969-c5b7eb33eb53\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf"
Oct 29 22:51:51.833673 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:51.833630    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2032f74c-64e7-4c12-a969-c5b7eb33eb53-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf\" (UID: \"2032f74c-64e7-4c12-a969-c5b7eb33eb53\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf"
Oct 29 22:51:51.833912 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:51.833692    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr7ds\" (UniqueName: \"kubernetes.io/projected/2032f74c-64e7-4c12-a969-c5b7eb33eb53-kube-api-access-zr7ds\") pod \"payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf\" (UID: \"2032f74c-64e7-4c12-a969-c5b7eb33eb53\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf"
Oct 29 22:51:51.834444 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:51.834419    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2032f74c-64e7-4c12-a969-c5b7eb33eb53-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf\" (UID: \"2032f74c-64e7-4c12-a969-c5b7eb33eb53\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf"
Oct 29 22:51:51.855168 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:51.855141    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr7ds\" (UniqueName: \"kubernetes.io/projected/2032f74c-64e7-4c12-a969-c5b7eb33eb53-kube-api-access-zr7ds\") pod \"payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf\" (UID: \"2032f74c-64e7-4c12-a969-c5b7eb33eb53\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf"
Oct 29 22:51:51.939260 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:51.939226    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf"
Oct 29 22:51:52.305310 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:52.303769    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf"]
Oct 29 22:51:52.583887 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:52.583763    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:52.632601 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:52.632563    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf" event={"ID":"2032f74c-64e7-4c12-a969-c5b7eb33eb53","Type":"ContainerStarted","Data":"cf840fcb64477dc0bf276c04b2e55292d05ba147d9b3fff5a21089ccd3b7ef1d"}
Oct 29 22:51:52.632744 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:52.632617    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf" event={"ID":"2032f74c-64e7-4c12-a969-c5b7eb33eb53","Type":"ContainerStarted","Data":"364e94cb31bb3eaf93cba75d7af62307820e1c7bd7b56e5b414d0c05ff087d9a"}
Oct 29 22:51:52.632831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:52.632741    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf"
Oct 29 22:51:52.702472 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:52.702440    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:52.811170 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:52.811125    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:53.584857 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:53.584828    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8"
Oct 29 22:51:53.626636 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:53.626567    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf" podStartSLOduration=2.62654801 podStartE2EDuration="2.62654801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:52.685728338 +0000 UTC m=+516.028677103" watchObservedRunningTime="2026-10-29 22:51:53.62654801 +0000 UTC m=+516.969496756"
Oct 29 22:51:54.437436 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.437401    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"]
Oct 29 22:51:54.441912 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.441886    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"
Oct 29 22:51:54.487208 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.485342    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:51:54.487208 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.485492    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pr5wj\""
Oct 29 22:51:54.487208 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.485578    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:51:54.487208 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.485661    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-e57216-serving-cert\""
Oct 29 22:51:54.487208 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.485740    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 22:51:54.489147 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.489119    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"]
Oct 29 22:51:54.584891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.584287    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp"
Oct 29 22:51:54.584891 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.584649    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg"
Oct 29 22:51:54.587861 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.587793    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dae43c-236e-4510-a6b9-0ffa73efe69f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8\" (UID: \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"
Oct 29 22:51:54.588000 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.587899    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dae43c-236e-4510-a6b9-0ffa73efe69f-metrics-certs\") pod \"maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8\" (UID: \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"
Oct 29 22:51:54.588000 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.587939    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49vl5\" (UniqueName: \"kubernetes.io/projected/c1dae43c-236e-4510-a6b9-0ffa73efe69f-kube-api-access-49vl5\") pod \"maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8\" (UID: \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"
Oct 29 22:51:54.588142 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.588119    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dae43c-236e-4510-a6b9-0ffa73efe69f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8\" (UID: \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"
Oct 29 22:51:54.598379 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.598345    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:54.691730 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.691642    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dae43c-236e-4510-a6b9-0ffa73efe69f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8\" (UID: \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"
Oct 29 22:51:54.691900 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.691750    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dae43c-236e-4510-a6b9-0ffa73efe69f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8\" (UID: \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"
Oct 29 22:51:54.691900 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.691836    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dae43c-236e-4510-a6b9-0ffa73efe69f-metrics-certs\") pod \"maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8\" (UID: \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"
Oct 29 22:51:54.691900 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.691893    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49vl5\" (UniqueName: \"kubernetes.io/projected/c1dae43c-236e-4510-a6b9-0ffa73efe69f-kube-api-access-49vl5\") pod \"maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8\" (UID: \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"
Oct 29 22:51:54.693654 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.693625    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dae43c-236e-4510-a6b9-0ffa73efe69f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8\" (UID: \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"
Oct 29 22:51:54.695522 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.695491    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dae43c-236e-4510-a6b9-0ffa73efe69f-metrics-certs\") pod \"maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8\" (UID: \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"
Oct 29 22:51:54.695935 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.695915    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dae43c-236e-4510-a6b9-0ffa73efe69f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8\" (UID: \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"
Oct 29 22:51:54.713494 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.713466    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49vl5\" (UniqueName: \"kubernetes.io/projected/c1dae43c-236e-4510-a6b9-0ffa73efe69f-kube-api-access-49vl5\") pod \"maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8\" (UID: \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"
Oct 29 22:51:54.714723 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.714688    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:54.755330 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.755304    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"
Oct 29 22:51:54.835684 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:54.835630    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:55.101105 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:55.100902    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"]
Oct 29 22:51:55.103873 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:51:55.103841    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1dae43c_236e_4510_a6b9_0ffa73efe69f.slice/crio-bd4c1a4c1452fcf08a0dbc94f3c20a384441cc149c3d2a6f693aa824d656578b WatchSource:0}: Error finding container bd4c1a4c1452fcf08a0dbc94f3c20a384441cc149c3d2a6f693aa824d656578b: Status 404 returned error can't find the container with id bd4c1a4c1452fcf08a0dbc94f3c20a384441cc149c3d2a6f693aa824d656578b
Oct 29 22:51:55.658215 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:55.658181    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8" event={"ID":"c1dae43c-236e-4510-a6b9-0ffa73efe69f","Type":"ContainerStarted","Data":"bd4c1a4c1452fcf08a0dbc94f3c20a384441cc149c3d2a6f693aa824d656578b"}
Oct 29 22:51:56.475276 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:56.475243    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:57.688876 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:57.688741    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8" event={"ID":"c1dae43c-236e-4510-a6b9-0ffa73efe69f","Type":"ContainerStarted","Data":"ed2771edf8dd43d39cac77fd9fa7370bf6c1ff3c1a57547c8a057bf1792e60a8"}
Oct 29 22:51:57.689383 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:57.689000    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"
Oct 29 22:51:57.730764 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:57.730711    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8" podStartSLOduration=1.7223109650000001 podStartE2EDuration="3.73069596s" totalImagesPullingTime="2.008384995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:57.7298825 +0000 UTC m=+521.072831245" watchObservedRunningTime="2026-10-29 22:51:57.73069596 +0000 UTC m=+521.073644705"
Oct 29 22:51:58.250687 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:58.250646    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:58.609911 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:58.609838    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r"
Oct 29 22:51:59.615927 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:51:59.615895    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs"
Oct 29 22:52:00.604111 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:00.604032    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:00.715325 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:00.715238    5428 generic.go:356] "Generic (PLEG): container finished" podID="887f4003-a75f-4799-8c46-2460a9f89f8e" containerID="c3d0d38ef97f1c8e80dae32b60b12317e232b9dbc412c47fad567d55f5e72c55" exitCode=0
Oct 29 22:52:00.715789 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:00.715368    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs" event={"ID":"887f4003-a75f-4799-8c46-2460a9f89f8e","Type":"ContainerDied","Data":"c3d0d38ef97f1c8e80dae32b60b12317e232b9dbc412c47fad567d55f5e72c55"}
Oct 29 22:52:00.715913 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:00.715849    5428 scope.go:122] "RemoveContainer" containerID="c3d0d38ef97f1c8e80dae32b60b12317e232b9dbc412c47fad567d55f5e72c55"
Oct 29 22:52:00.717184 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:00.717153    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:00.717268 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:00.717224    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:00.717268 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:00.717258    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:01.728202 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:01.728161    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs" event={"ID":"887f4003-a75f-4799-8c46-2460a9f89f8e","Type":"ContainerStarted","Data":"7e85b8a2e5e5c54aabe23ffde9aad982fd57b63cb28865eae3983408e10cf321"}
Oct 29 22:52:01.988327 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:01.988232    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:52:01.988653 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:01.988631    5428 patch_prober.go:29] interesting pod/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:52:01.988820 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:01.988774    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs" podUID="887f4003-a75f-4799-8c46-2460a9f89f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 22:52:02.828431 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:02.828387    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:03.013850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.000928    5428 patch_prober.go:29] interesting pod/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:52:03.013850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.001004    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs" podUID="887f4003-a75f-4799-8c46-2460a9f89f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 22:52:03.015491 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.015450    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:03.130225 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.130194    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:03.289744 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.289700    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj"]
Oct 29 22:52:03.297582 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.297557    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj"
Oct 29 22:52:03.300490 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.300465    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj"]
Oct 29 22:52:03.312255 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.312210    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:52:03.322512 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.322457    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\""
Oct 29 22:52:03.340301 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.340261    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbe51a7-678d-4028-872f-02ea1463723e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e406071d-g9jfj\" (UID: \"bbbe51a7-678d-4028-872f-02ea1463723e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj"
Oct 29 22:52:03.340484 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.340454    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9zvn\" (UniqueName: \"kubernetes.io/projected/bbbe51a7-678d-4028-872f-02ea1463723e-kube-api-access-k9zvn\") pod \"maas-api-revoke-keys-e2e-derive-e406071d-g9jfj\" (UID: \"bbbe51a7-678d-4028-872f-02ea1463723e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj"
Oct 29 22:52:03.340529 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.340514    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbe51a7-678d-4028-872f-02ea1463723e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e406071d-g9jfj\" (UID: \"bbbe51a7-678d-4028-872f-02ea1463723e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj"
Oct 29 22:52:03.441459 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.441372    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9zvn\" (UniqueName: \"kubernetes.io/projected/bbbe51a7-678d-4028-872f-02ea1463723e-kube-api-access-k9zvn\") pod \"maas-api-revoke-keys-e2e-derive-e406071d-g9jfj\" (UID: \"bbbe51a7-678d-4028-872f-02ea1463723e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj"
Oct 29 22:52:03.441459 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.441430    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbe51a7-678d-4028-872f-02ea1463723e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e406071d-g9jfj\" (UID: \"bbbe51a7-678d-4028-872f-02ea1463723e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj"
Oct 29 22:52:03.441685 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.441550    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbe51a7-678d-4028-872f-02ea1463723e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e406071d-g9jfj\" (UID: \"bbbe51a7-678d-4028-872f-02ea1463723e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj"
Oct 29 22:52:03.442343 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.442312    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbe51a7-678d-4028-872f-02ea1463723e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e406071d-g9jfj\" (UID: \"bbbe51a7-678d-4028-872f-02ea1463723e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj"
Oct 29 22:52:03.444050 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.444025    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbe51a7-678d-4028-872f-02ea1463723e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e406071d-g9jfj\" (UID: \"bbbe51a7-678d-4028-872f-02ea1463723e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj"
Oct 29 22:52:03.492606 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.489387    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9zvn\" (UniqueName: \"kubernetes.io/projected/bbbe51a7-678d-4028-872f-02ea1463723e-kube-api-access-k9zvn\") pod \"maas-api-revoke-keys-e2e-derive-e406071d-g9jfj\" (UID: \"bbbe51a7-678d-4028-872f-02ea1463723e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj"
Oct 29 22:52:03.624644 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.624610    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj"
Oct 29 22:52:03.650060 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.650035    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf"
Oct 29 22:52:03.709771 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.708824    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"
Oct 29 22:52:03.988795 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.988305    5428 patch_prober.go:29] interesting pod/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:52:03.988795 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.988374    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs" podUID="887f4003-a75f-4799-8c46-2460a9f89f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 22:52:04.000862 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:03.995831    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj"]
Oct 29 22:52:04.758740 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:04.758702    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj" event={"ID":"bbbe51a7-678d-4028-872f-02ea1463723e","Type":"ContainerStarted","Data":"08464d986355e759de90a0accab1eb2b793d29eebd8214d794e520d6724b6525"}
Oct 29 22:52:04.902981 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:04.902943    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:04.986753 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:04.986302    5428 patch_prober.go:29] interesting pod/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:52:04.986753 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:04.986363    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs" podUID="887f4003-a75f-4799-8c46-2460a9f89f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 22:52:05.986389 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:05.985927    5428 patch_prober.go:29] interesting pod/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:52:05.986389 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:05.985990    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs" podUID="887f4003-a75f-4799-8c46-2460a9f89f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 22:52:06.797560 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:06.797510    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj" event={"ID":"bbbe51a7-678d-4028-872f-02ea1463723e","Type":"ContainerStarted","Data":"476b2a838007749fe4875e03a56f07b0c56cfef438771702cea7359a7515dcdc"}
Oct 29 22:52:06.835744 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:06.835690    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj" podStartSLOduration=1.494007598 podStartE2EDuration="3.835674812s" totalImagesPullingTime="2.341667214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:06.833623075 +0000 UTC m=+530.176571833" watchObservedRunningTime="2026-10-29 22:52:06.835674812 +0000 UTC m=+530.178623556"
Oct 29 22:52:06.896287 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:06.896233    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:06.986397 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:06.986307    5428 patch_prober.go:29] interesting pod/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:52:06.986660 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:06.986628    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs" podUID="887f4003-a75f-4799-8c46-2460a9f89f8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 29 22:52:07.805626 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:07.805536    5428 generic.go:356] "Generic (PLEG): container finished" podID="bbbe51a7-678d-4028-872f-02ea1463723e" containerID="476b2a838007749fe4875e03a56f07b0c56cfef438771702cea7359a7515dcdc" exitCode=0
Oct 29 22:52:07.805851 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:07.805624    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj" event={"ID":"bbbe51a7-678d-4028-872f-02ea1463723e","Type":"ContainerDied","Data":"476b2a838007749fe4875e03a56f07b0c56cfef438771702cea7359a7515dcdc"}
Oct 29 22:52:07.807537 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:07.807512    5428 generic.go:356] "Generic (PLEG): container finished" podID="fd186355-c594-4b2b-b716-21ced7841757" containerID="d0cb063f35742e4738e0e3fb88629419b88866f7719cdf487f10ec82e54f5e2d" exitCode=0
Oct 29 22:52:07.807635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:07.807579    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn" event={"ID":"fd186355-c594-4b2b-b716-21ced7841757","Type":"ContainerDied","Data":"d0cb063f35742e4738e0e3fb88629419b88866f7719cdf487f10ec82e54f5e2d"}
Oct 29 22:52:07.807967 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:07.807952    5428 scope.go:122] "RemoveContainer" containerID="d0cb063f35742e4738e0e3fb88629419b88866f7719cdf487f10ec82e54f5e2d"
Oct 29 22:52:07.809263 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:07.809237    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:07.809368 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:07.809316    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:07.809434 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:07.809363    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:07.989545 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:07.989511    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:52:07.990627 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:07.990606    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:52:07.991277 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:07.991254    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:52:08.513554 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:08.513519    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:52:08.819107 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:08.819006    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn" event={"ID":"fd186355-c594-4b2b-b716-21ced7841757","Type":"ContainerStarted","Data":"e6297090865be69d90f35aa6c8ed2107c2763f1b707043510109558e9740ef0a"}
Oct 29 22:52:09.011436 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.011412    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj"
Oct 29 22:52:09.032939 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.032906    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:09.089346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.089252    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbe51a7-678d-4028-872f-02ea1463723e-kube-api-access-k9zvn\" (UniqueName: \"kubernetes.io/projected/bbbe51a7-678d-4028-872f-02ea1463723e-kube-api-access-k9zvn\") pod \"bbbe51a7-678d-4028-872f-02ea1463723e\" (UID: \"bbbe51a7-678d-4028-872f-02ea1463723e\") "
Oct 29 22:52:09.089998 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.089969    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbbe51a7-678d-4028-872f-02ea1463723e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbe51a7-678d-4028-872f-02ea1463723e-maas-api-xks-ca\") pod \"bbbe51a7-678d-4028-872f-02ea1463723e\" (UID: \"bbbe51a7-678d-4028-872f-02ea1463723e\") "
Oct 29 22:52:09.091579 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.091554    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbe51a7-678d-4028-872f-02ea1463723e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbe51a7-678d-4028-872f-02ea1463723e-maas-api-openshift-ca\") pod \"bbbe51a7-678d-4028-872f-02ea1463723e\" (UID: \"bbbe51a7-678d-4028-872f-02ea1463723e\") "
Oct 29 22:52:09.092692 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.092664    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbe51a7-678d-4028-872f-02ea1463723e-maas-api-openshift-ca" pod "bbbe51a7-678d-4028-872f-02ea1463723e" (UID: "bbbe51a7-678d-4028-872f-02ea1463723e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:52:09.102521 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.102466    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbbe51a7-678d-4028-872f-02ea1463723e-maas-api-xks-ca" pod "bbbe51a7-678d-4028-872f-02ea1463723e" (UID: "bbbe51a7-678d-4028-872f-02ea1463723e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:52:09.104794 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.104758    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbe51a7-678d-4028-872f-02ea1463723e-kube-api-access-k9zvn" pod "bbbe51a7-678d-4028-872f-02ea1463723e" (UID: "bbbe51a7-678d-4028-872f-02ea1463723e"). InnerVolumeSpecName "kube-api-access-k9zvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:09.193578 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.193541    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9zvn\" (UniqueName: \"kubernetes.io/projected/bbbe51a7-678d-4028-872f-02ea1463723e-kube-api-access-k9zvn\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:09.193578 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.193569    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbe51a7-678d-4028-872f-02ea1463723e-maas-api-xks-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:09.193578 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.193579    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbe51a7-678d-4028-872f-02ea1463723e-maas-api-openshift-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:09.285074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.285029    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8"]
Oct 29 22:52:09.305016 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.304984    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st"]
Oct 29 22:52:09.416655 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.416621    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:09.527625 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.527588    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:09.586087 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.586049    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5"]
Oct 29 22:52:09.587117 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.587088    5428 cpu_manager.go:414] "containerMap: removing container" podUID="bbbe51a7-678d-4028-872f-02ea1463723e" containerName="revoke-keys"
Oct 29 22:52:09.587117 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.587113    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbe51a7-678d-4028-872f-02ea1463723e" containerName="revoke-keys"
Oct 29 22:52:09.587289 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.587251    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbe51a7-678d-4028-872f-02ea1463723e" containerName="revoke-keys"
Oct 29 22:52:09.591463 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.591441    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5"
Oct 29 22:52:09.609049 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.609021    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5"]
Oct 29 22:52:09.629339 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.629311    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:52:09.630416 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.630389    5428 patch_prober.go:29] interesting pod/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:52:09.630530 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.630443    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn" podUID="fd186355-c594-4b2b-b716-21ced7841757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 22:52:09.700287 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.700211    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jxzt\" (UniqueName: \"kubernetes.io/projected/cfac23f3-7a49-4e44-8123-cf9a0286174d-kube-api-access-5jxzt\") pod \"maas-api-revoke-keys-e2e-ait-865d946f-l6bl5\" (UID: \"cfac23f3-7a49-4e44-8123-cf9a0286174d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5"
Oct 29 22:52:09.700287 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.700257    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfac23f3-7a49-4e44-8123-cf9a0286174d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-865d946f-l6bl5\" (UID: \"cfac23f3-7a49-4e44-8123-cf9a0286174d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5"
Oct 29 22:52:09.700450 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.700382    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfac23f3-7a49-4e44-8123-cf9a0286174d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-865d946f-l6bl5\" (UID: \"cfac23f3-7a49-4e44-8123-cf9a0286174d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5"
Oct 29 22:52:09.801162 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.801125    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jxzt\" (UniqueName: \"kubernetes.io/projected/cfac23f3-7a49-4e44-8123-cf9a0286174d-kube-api-access-5jxzt\") pod \"maas-api-revoke-keys-e2e-ait-865d946f-l6bl5\" (UID: \"cfac23f3-7a49-4e44-8123-cf9a0286174d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5"
Oct 29 22:52:09.801325 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.801181    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfac23f3-7a49-4e44-8123-cf9a0286174d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-865d946f-l6bl5\" (UID: \"cfac23f3-7a49-4e44-8123-cf9a0286174d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5"
Oct 29 22:52:09.801325 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.801214    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfac23f3-7a49-4e44-8123-cf9a0286174d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-865d946f-l6bl5\" (UID: \"cfac23f3-7a49-4e44-8123-cf9a0286174d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5"
Oct 29 22:52:09.801870 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.801845    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfac23f3-7a49-4e44-8123-cf9a0286174d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-865d946f-l6bl5\" (UID: \"cfac23f3-7a49-4e44-8123-cf9a0286174d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5"
Oct 29 22:52:09.803575 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.803555    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfac23f3-7a49-4e44-8123-cf9a0286174d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-865d946f-l6bl5\" (UID: \"cfac23f3-7a49-4e44-8123-cf9a0286174d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5"
Oct 29 22:52:09.822635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.822611    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jxzt\" (UniqueName: \"kubernetes.io/projected/cfac23f3-7a49-4e44-8123-cf9a0286174d-kube-api-access-5jxzt\") pod \"maas-api-revoke-keys-e2e-ait-865d946f-l6bl5\" (UID: \"cfac23f3-7a49-4e44-8123-cf9a0286174d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5"
Oct 29 22:52:09.826183 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.826162    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj"
Oct 29 22:52:09.826306 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.826191    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj" event={"ID":"bbbe51a7-678d-4028-872f-02ea1463723e","Type":"ContainerDied","Data":"08464d986355e759de90a0accab1eb2b793d29eebd8214d794e520d6724b6525"}
Oct 29 22:52:09.826306 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.826247    5428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08464d986355e759de90a0accab1eb2b793d29eebd8214d794e520d6724b6525"
Oct 29 22:52:09.909869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:09.909571    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5"
Oct 29 22:52:10.250494 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:10.250470    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5"]
Oct 29 22:52:10.252990 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:52:10.252952    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfac23f3_7a49_4e44_8123_cf9a0286174d.slice/crio-fbbdbd56824e1cc0445cc239cacafa48ad6d7da542fff453db4c7f711dd77d21 WatchSource:0}: Error finding container fbbdbd56824e1cc0445cc239cacafa48ad6d7da542fff453db4c7f711dd77d21: Status 404 returned error can't find the container with id fbbdbd56824e1cc0445cc239cacafa48ad6d7da542fff453db4c7f711dd77d21
Oct 29 22:52:10.256238 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:10.256214    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:10.389329 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:10.389289    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:10.630314 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:10.630274    5428 patch_prober.go:29] interesting pod/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:52:10.630482 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:10.630329    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn" podUID="fd186355-c594-4b2b-b716-21ced7841757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 22:52:10.845127 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:10.844964    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5" event={"ID":"cfac23f3-7a49-4e44-8123-cf9a0286174d","Type":"ContainerStarted","Data":"3d8f14cdf8569f4e26ba97511266a981b6549d5d9d5ecb1af858c716f5f18e1b"}
Oct 29 22:52:10.845127 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:10.845024    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5" event={"ID":"cfac23f3-7a49-4e44-8123-cf9a0286174d","Type":"ContainerStarted","Data":"fbbdbd56824e1cc0445cc239cacafa48ad6d7da542fff453db4c7f711dd77d21"}
Oct 29 22:52:10.897531 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:10.897466    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5" podStartSLOduration=1.897451474 podStartE2EDuration="1.897451474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:10.89632921 +0000 UTC m=+534.239277955" watchObservedRunningTime="2026-10-29 22:52:10.897451474 +0000 UTC m=+534.240400223"
Oct 29 22:52:11.258196 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:11.258160    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:11.630405 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:11.630370    5428 patch_prober.go:29] interesting pod/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:52:11.630548 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:11.630425    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn" podUID="fd186355-c594-4b2b-b716-21ced7841757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 22:52:11.851653 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:11.851618    5428 generic.go:356] "Generic (PLEG): container finished" podID="cfac23f3-7a49-4e44-8123-cf9a0286174d" containerID="3d8f14cdf8569f4e26ba97511266a981b6549d5d9d5ecb1af858c716f5f18e1b" exitCode=0
Oct 29 22:52:11.851812 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:11.851677    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5" event={"ID":"cfac23f3-7a49-4e44-8123-cf9a0286174d","Type":"ContainerDied","Data":"3d8f14cdf8569f4e26ba97511266a981b6549d5d9d5ecb1af858c716f5f18e1b"}
Oct 29 22:52:12.630121 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:12.630075    5428 patch_prober.go:29] interesting pod/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:52:12.630518 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:12.630138    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn" podUID="fd186355-c594-4b2b-b716-21ced7841757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 22:52:12.823446 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:12.823409    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:13.095513 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.095482    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5"
Oct 29 22:52:13.140697 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.140658    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfac23f3-7a49-4e44-8123-cf9a0286174d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfac23f3-7a49-4e44-8123-cf9a0286174d-maas-api-openshift-ca\") pod \"cfac23f3-7a49-4e44-8123-cf9a0286174d\" (UID: \"cfac23f3-7a49-4e44-8123-cf9a0286174d\") "
Oct 29 22:52:13.140889 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.140736    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfac23f3-7a49-4e44-8123-cf9a0286174d-kube-api-access-5jxzt\" (UniqueName: \"kubernetes.io/projected/cfac23f3-7a49-4e44-8123-cf9a0286174d-kube-api-access-5jxzt\") pod \"cfac23f3-7a49-4e44-8123-cf9a0286174d\" (UID: \"cfac23f3-7a49-4e44-8123-cf9a0286174d\") "
Oct 29 22:52:13.140889 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.140774    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfac23f3-7a49-4e44-8123-cf9a0286174d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfac23f3-7a49-4e44-8123-cf9a0286174d-maas-api-xks-ca\") pod \"cfac23f3-7a49-4e44-8123-cf9a0286174d\" (UID: \"cfac23f3-7a49-4e44-8123-cf9a0286174d\") "
Oct 29 22:52:13.141075 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.141045    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfac23f3-7a49-4e44-8123-cf9a0286174d-maas-api-openshift-ca" pod "cfac23f3-7a49-4e44-8123-cf9a0286174d" (UID: "cfac23f3-7a49-4e44-8123-cf9a0286174d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:52:13.141349 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.141323    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfac23f3-7a49-4e44-8123-cf9a0286174d-maas-api-openshift-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:13.143033 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.142995    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfac23f3-7a49-4e44-8123-cf9a0286174d-maas-api-xks-ca" pod "cfac23f3-7a49-4e44-8123-cf9a0286174d" (UID: "cfac23f3-7a49-4e44-8123-cf9a0286174d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:52:13.143393 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.143365    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfac23f3-7a49-4e44-8123-cf9a0286174d-kube-api-access-5jxzt" pod "cfac23f3-7a49-4e44-8123-cf9a0286174d" (UID: "cfac23f3-7a49-4e44-8123-cf9a0286174d"). InnerVolumeSpecName "kube-api-access-5jxzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:13.242045 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.241967    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jxzt\" (UniqueName: \"kubernetes.io/projected/cfac23f3-7a49-4e44-8123-cf9a0286174d-kube-api-access-5jxzt\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:13.242045 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.241999    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfac23f3-7a49-4e44-8123-cf9a0286174d-maas-api-xks-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:13.386106 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.386063    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:13.500648 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.500545    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:13.608175 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.608144    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:13.630160 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.630124    5428 patch_prober.go:29] interesting pod/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:52:13.630566 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.630177    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn" podUID="fd186355-c594-4b2b-b716-21ced7841757" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 29 22:52:13.785535 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.785449    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:13.877037 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.876694    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5" event={"ID":"cfac23f3-7a49-4e44-8123-cf9a0286174d","Type":"ContainerDied","Data":"fbbdbd56824e1cc0445cc239cacafa48ad6d7da542fff453db4c7f711dd77d21"}
Oct 29 22:52:13.877037 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.876763    5428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbbdbd56824e1cc0445cc239cacafa48ad6d7da542fff453db4c7f711dd77d21"
Oct 29 22:52:13.877037 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:13.876861    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5"
Oct 29 22:52:14.208701 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.208669    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp"]
Oct 29 22:52:14.285988 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.285942    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8" podUID="f706efa1-73c8-4a34-a0b4-7be82c2be566" containerName="payload-processing" containerID="cri-o://c22a5370913c369b0252c9687414da250213777a5dc5845f1bf50706a3f355c8" gracePeriod=25
Oct 29 22:52:14.306027 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.305979    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st" podUID="e4dfd4c3-c1a6-4438-a299-d006b1107e35" containerName="payload-pre-processing" containerID="cri-o://e23544037321ac7bec6e627178c437327a8447da350ded05a9ed06c6fe8a8330" gracePeriod=25
Oct 29 22:52:14.354509 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.354479    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:14.477768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.477692    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:14.578225 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.578201    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8"
Oct 29 22:52:14.598951 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.598926    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:14.606060 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.606038    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st"
Oct 29 22:52:14.634441 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.634415    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:52:14.634834 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.634717    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:52:14.635548 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.635528    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:52:14.658835 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.658784    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4dfd4c3-c1a6-4438-a299-d006b1107e35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4dfd4c3-c1a6-4438-a299-d006b1107e35-plugins-config-volume\") pod \"e4dfd4c3-c1a6-4438-a299-d006b1107e35\" (UID: \"e4dfd4c3-c1a6-4438-a299-d006b1107e35\") "
Oct 29 22:52:14.658998 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.658920    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4dfd4c3-c1a6-4438-a299-d006b1107e35-kube-api-access-nsg4n\" (UniqueName: \"kubernetes.io/projected/e4dfd4c3-c1a6-4438-a299-d006b1107e35-kube-api-access-nsg4n\") pod \"e4dfd4c3-c1a6-4438-a299-d006b1107e35\" (UID: \"e4dfd4c3-c1a6-4438-a299-d006b1107e35\") "
Oct 29 22:52:14.658998 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.658951    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f706efa1-73c8-4a34-a0b4-7be82c2be566-kube-api-access-4bwxt\" (UniqueName: \"kubernetes.io/projected/f706efa1-73c8-4a34-a0b4-7be82c2be566-kube-api-access-4bwxt\") pod \"f706efa1-73c8-4a34-a0b4-7be82c2be566\" (UID: \"f706efa1-73c8-4a34-a0b4-7be82c2be566\") "
Oct 29 22:52:14.658998 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.658983    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f706efa1-73c8-4a34-a0b4-7be82c2be566-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f706efa1-73c8-4a34-a0b4-7be82c2be566-plugins-config-volume\") pod \"f706efa1-73c8-4a34-a0b4-7be82c2be566\" (UID: \"f706efa1-73c8-4a34-a0b4-7be82c2be566\") "
Oct 29 22:52:14.659336 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.659195    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4dfd4c3-c1a6-4438-a299-d006b1107e35-plugins-config-volume" pod "e4dfd4c3-c1a6-4438-a299-d006b1107e35" (UID: "e4dfd4c3-c1a6-4438-a299-d006b1107e35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:52:14.659496 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.659396    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f706efa1-73c8-4a34-a0b4-7be82c2be566-plugins-config-volume" pod "f706efa1-73c8-4a34-a0b4-7be82c2be566" (UID: "f706efa1-73c8-4a34-a0b4-7be82c2be566"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:52:14.659840 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.659819    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4dfd4c3-c1a6-4438-a299-d006b1107e35-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:14.659915 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.659844    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f706efa1-73c8-4a34-a0b4-7be82c2be566-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:14.661365 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.661336    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f706efa1-73c8-4a34-a0b4-7be82c2be566-kube-api-access-4bwxt" pod "f706efa1-73c8-4a34-a0b4-7be82c2be566" (UID: "f706efa1-73c8-4a34-a0b4-7be82c2be566"). InnerVolumeSpecName "kube-api-access-4bwxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:14.661493 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.661474    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4dfd4c3-c1a6-4438-a299-d006b1107e35-kube-api-access-nsg4n" pod "e4dfd4c3-c1a6-4438-a299-d006b1107e35" (UID: "e4dfd4c3-c1a6-4438-a299-d006b1107e35"). InnerVolumeSpecName "kube-api-access-nsg4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:14.761088 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.761004    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsg4n\" (UniqueName: \"kubernetes.io/projected/e4dfd4c3-c1a6-4438-a299-d006b1107e35-kube-api-access-nsg4n\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:14.761088 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.761034    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bwxt\" (UniqueName: \"kubernetes.io/projected/f706efa1-73c8-4a34-a0b4-7be82c2be566-kube-api-access-4bwxt\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:14.885506 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.885473    5428 generic.go:356] "Generic (PLEG): container finished" podID="e4dfd4c3-c1a6-4438-a299-d006b1107e35" containerID="e23544037321ac7bec6e627178c437327a8447da350ded05a9ed06c6fe8a8330" exitCode=0
Oct 29 22:52:14.885705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.885538    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st"
Oct 29 22:52:14.885705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.885584    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st" event={"ID":"e4dfd4c3-c1a6-4438-a299-d006b1107e35","Type":"ContainerDied","Data":"e23544037321ac7bec6e627178c437327a8447da350ded05a9ed06c6fe8a8330"}
Oct 29 22:52:14.885705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.885626    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st" event={"ID":"e4dfd4c3-c1a6-4438-a299-d006b1107e35","Type":"ContainerDied","Data":"e967d7cf5bc47babafb2bc5079f94c0f3b2251f32106b7483b021d03d107f2dc"}
Oct 29 22:52:14.885705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.885648    5428 scope.go:122] "RemoveContainer" containerID="e23544037321ac7bec6e627178c437327a8447da350ded05a9ed06c6fe8a8330"
Oct 29 22:52:14.887860 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.887609    5428 generic.go:356] "Generic (PLEG): container finished" podID="f706efa1-73c8-4a34-a0b4-7be82c2be566" containerID="c22a5370913c369b0252c9687414da250213777a5dc5845f1bf50706a3f355c8" exitCode=0
Oct 29 22:52:14.887860 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.887662    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8"
Oct 29 22:52:14.887860 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.887720    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8" event={"ID":"f706efa1-73c8-4a34-a0b4-7be82c2be566","Type":"ContainerDied","Data":"c22a5370913c369b0252c9687414da250213777a5dc5845f1bf50706a3f355c8"}
Oct 29 22:52:14.887860 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.887791    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8" event={"ID":"f706efa1-73c8-4a34-a0b4-7be82c2be566","Type":"ContainerDied","Data":"1e296b9a4f2eec5adc08a98451f922d4310c7669194f2a4e104352c74b7ba6c2"}
Oct 29 22:52:14.896148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.896128    5428 scope.go:122] "RemoveContainer" containerID="e23544037321ac7bec6e627178c437327a8447da350ded05a9ed06c6fe8a8330"
Oct 29 22:52:14.896527 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:52:14.896508    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e23544037321ac7bec6e627178c437327a8447da350ded05a9ed06c6fe8a8330\": container with ID starting with e23544037321ac7bec6e627178c437327a8447da350ded05a9ed06c6fe8a8330 not found: ID does not exist" containerID="e23544037321ac7bec6e627178c437327a8447da350ded05a9ed06c6fe8a8330"
Oct 29 22:52:14.896585 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.896536    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e23544037321ac7bec6e627178c437327a8447da350ded05a9ed06c6fe8a8330"} err="failed to get container status \"e23544037321ac7bec6e627178c437327a8447da350ded05a9ed06c6fe8a8330\": rpc error: code = NotFound desc = could not find container \"e23544037321ac7bec6e627178c437327a8447da350ded05a9ed06c6fe8a8330\": container with ID starting with e23544037321ac7bec6e627178c437327a8447da350ded05a9ed06c6fe8a8330 not found: ID does not exist"
Oct 29 22:52:14.896585 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.896553    5428 scope.go:122] "RemoveContainer" containerID="c22a5370913c369b0252c9687414da250213777a5dc5845f1bf50706a3f355c8"
Oct 29 22:52:14.907870 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.907847    5428 scope.go:122] "RemoveContainer" containerID="c22a5370913c369b0252c9687414da250213777a5dc5845f1bf50706a3f355c8"
Oct 29 22:52:14.908791 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:52:14.908764    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c22a5370913c369b0252c9687414da250213777a5dc5845f1bf50706a3f355c8\": container with ID starting with c22a5370913c369b0252c9687414da250213777a5dc5845f1bf50706a3f355c8 not found: ID does not exist" containerID="c22a5370913c369b0252c9687414da250213777a5dc5845f1bf50706a3f355c8"
Oct 29 22:52:14.909072 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.909049    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22a5370913c369b0252c9687414da250213777a5dc5845f1bf50706a3f355c8"} err="failed to get container status \"c22a5370913c369b0252c9687414da250213777a5dc5845f1bf50706a3f355c8\": rpc error: code = NotFound desc = could not find container \"c22a5370913c369b0252c9687414da250213777a5dc5845f1bf50706a3f355c8\": container with ID starting with c22a5370913c369b0252c9687414da250213777a5dc5845f1bf50706a3f355c8 not found: ID does not exist"
Oct 29 22:52:14.935142 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.935107    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8"]
Oct 29 22:52:14.944934 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.944901    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5db969e0-6c4cd5bf6b-sh9k8"]
Oct 29 22:52:14.976999 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.976966    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st"]
Oct 29 22:52:14.986260 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:14.986222    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5db969e0-6878b4984c-bk8st"]
Oct 29 22:52:15.010875 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.010793    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"]
Oct 29 22:52:15.011710 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.011633    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e4dfd4c3-c1a6-4438-a299-d006b1107e35" containerName="payload-pre-processing"
Oct 29 22:52:15.011710 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.011660    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4dfd4c3-c1a6-4438-a299-d006b1107e35" containerName="payload-pre-processing"
Oct 29 22:52:15.011710 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.011690    5428 cpu_manager.go:414] "containerMap: removing container" podUID="cfac23f3-7a49-4e44-8123-cf9a0286174d" containerName="revoke-keys"
Oct 29 22:52:15.011710 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.011699    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfac23f3-7a49-4e44-8123-cf9a0286174d" containerName="revoke-keys"
Oct 29 22:52:15.012062 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.011735    5428 cpu_manager.go:414] "containerMap: removing container" podUID="f706efa1-73c8-4a34-a0b4-7be82c2be566" containerName="payload-processing"
Oct 29 22:52:15.012062 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.011744    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f706efa1-73c8-4a34-a0b4-7be82c2be566" containerName="payload-processing"
Oct 29 22:52:15.012062 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.011890    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4dfd4c3-c1a6-4438-a299-d006b1107e35" containerName="payload-pre-processing"
Oct 29 22:52:15.012062 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.011907    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfac23f3-7a49-4e44-8123-cf9a0286174d" containerName="revoke-keys"
Oct 29 22:52:15.012062 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.011921    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f706efa1-73c8-4a34-a0b4-7be82c2be566" containerName="payload-processing"
Oct 29 22:52:15.017222 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.017195    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.037016 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.036841    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-32c192-openshift-default-dockercfg-m9mqd\""
Oct 29 22:52:15.041413 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.041383    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"]
Oct 29 22:52:15.066957 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.063177    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:15.066957 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.064213    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/751014a5-e82e-4403-83b7-f2e648514119-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.066957 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.064247    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.066957 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.064288    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.066957 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.064316    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.066957 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.064347    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.066957 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.064409    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.066957 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.064439    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bn7v\" (UniqueName: \"kubernetes.io/projected/751014a5-e82e-4403-83b7-f2e648514119-kube-api-access-6bn7v\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.066957 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.064483    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/751014a5-e82e-4403-83b7-f2e648514119-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.066957 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.064512    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/751014a5-e82e-4403-83b7-f2e648514119-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.165543 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.165507    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.165735 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.165573    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.165735 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.165660    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.165735 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.165705    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bn7v\" (UniqueName: \"kubernetes.io/projected/751014a5-e82e-4403-83b7-f2e648514119-kube-api-access-6bn7v\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.165933 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.165759    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/751014a5-e82e-4403-83b7-f2e648514119-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.165933 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.165833    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/751014a5-e82e-4403-83b7-f2e648514119-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.165933 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.165880    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/751014a5-e82e-4403-83b7-f2e648514119-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.165933 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.165919    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.166133 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.165973    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.166133 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.165988    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.166359 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.166336    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.166508 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.166417    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.166767 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.166739    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/751014a5-e82e-4403-83b7-f2e648514119-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.167035 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.167017    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.168839 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.168791    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.169082 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.169062    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/751014a5-e82e-4403-83b7-f2e648514119-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.187055 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.187029    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/751014a5-e82e-4403-83b7-f2e648514119-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.187164 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.187099    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bn7v\" (UniqueName: \"kubernetes.io/projected/751014a5-e82e-4403-83b7-f2e648514119-kube-api-access-6bn7v\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.207694 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.207667    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:15.346002 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.345970    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:15.464314 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.464267    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:15.666934 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.666903    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"]
Oct 29 22:52:15.669822 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:52:15.669777    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751014a5_e82e_4403_83b7_f2e648514119.slice/crio-2fc54d46dbcc7d0362917f24b18c3c902fb165e718df269c19fb5e76ec6e45bb WatchSource:0}: Error finding container 2fc54d46dbcc7d0362917f24b18c3c902fb165e718df269c19fb5e76ec6e45bb: Status 404 returned error can't find the container with id 2fc54d46dbcc7d0362917f24b18c3c902fb165e718df269c19fb5e76ec6e45bb
Oct 29 22:52:15.673018 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.672971    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:15.673162 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.673137    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:15.673257 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.673241    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:15.896029 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.895988    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh" event={"ID":"751014a5-e82e-4403-83b7-f2e648514119","Type":"ContainerStarted","Data":"80c7dcab7d14ea41d1fe5c5dcb6307b187bd8377b64b6d99b3c72e48e4a1b7d9"}
Oct 29 22:52:15.896201 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.896053    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh" event={"ID":"751014a5-e82e-4403-83b7-f2e648514119","Type":"ContainerStarted","Data":"2fc54d46dbcc7d0362917f24b18c3c902fb165e718df269c19fb5e76ec6e45bb"}
Oct 29 22:52:15.975693 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:15.975578    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh" podStartSLOduration=1.975564242 podStartE2EDuration="1.975564242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:15.970908329 +0000 UTC m=+539.313857073" watchObservedRunningTime="2026-10-29 22:52:15.975564242 +0000 UTC m=+539.318512987"
Oct 29 22:52:16.346357 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:16.346271    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:16.351328 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:16.351296    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:16.351949 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:16.351927    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:16.777002 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:16.776968    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4dfd4c3-c1a6-4438-a299-d006b1107e35" path="/var/lib/kubelet/pods/e4dfd4c3-c1a6-4438-a299-d006b1107e35/volumes"
Oct 29 22:52:16.777351 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:16.777326    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f706efa1-73c8-4a34-a0b4-7be82c2be566" path="/var/lib/kubelet/pods/f706efa1-73c8-4a34-a0b4-7be82c2be566/volumes"
Oct 29 22:52:16.904128 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:16.904098    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:16.905287 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:16.905261    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:16.964718 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:16.964691    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:17.118339 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:17.118307    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:17.254555 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:17.254523    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:17.366884 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:52:17.366850    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:52:17.487622 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:17.487500    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:18.500104 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:18.500066    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:18.634760 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:18.634728    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:19.215539 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.214785    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp" podUID="6d3f2e47-48c7-4691-938d-5424e634f2f5" containerName="payload-processing" containerID="cri-o://d82ee008b9bf72c12bac4b4b76a0db899d583d785ac5f7d54fb434326d496247" gracePeriod=25
Oct 29 22:52:19.398950 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.398910    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:19.520184 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.520155    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp"
Oct 29 22:52:19.530051 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.530022    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:19.619908 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.619873    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d3f2e47-48c7-4691-938d-5424e634f2f5-kube-api-access-nvnz7\" (UniqueName: \"kubernetes.io/projected/6d3f2e47-48c7-4691-938d-5424e634f2f5-kube-api-access-nvnz7\") pod \"6d3f2e47-48c7-4691-938d-5424e634f2f5\" (UID: \"6d3f2e47-48c7-4691-938d-5424e634f2f5\") "
Oct 29 22:52:19.620138 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.620056    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d3f2e47-48c7-4691-938d-5424e634f2f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3f2e47-48c7-4691-938d-5424e634f2f5-plugins-config-volume\") pod \"6d3f2e47-48c7-4691-938d-5424e634f2f5\" (UID: \"6d3f2e47-48c7-4691-938d-5424e634f2f5\") "
Oct 29 22:52:19.620421 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.620398    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3f2e47-48c7-4691-938d-5424e634f2f5-plugins-config-volume" pod "6d3f2e47-48c7-4691-938d-5424e634f2f5" (UID: "6d3f2e47-48c7-4691-938d-5424e634f2f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:52:19.620512 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.620493    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3f2e47-48c7-4691-938d-5424e634f2f5-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:19.622076 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.622048    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3f2e47-48c7-4691-938d-5424e634f2f5-kube-api-access-nvnz7" pod "6d3f2e47-48c7-4691-938d-5424e634f2f5" (UID: "6d3f2e47-48c7-4691-938d-5424e634f2f5"). InnerVolumeSpecName "kube-api-access-nvnz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:19.721283 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.721251    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvnz7\" (UniqueName: \"kubernetes.io/projected/6d3f2e47-48c7-4691-938d-5424e634f2f5-kube-api-access-nvnz7\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:19.924928 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.924885    5428 generic.go:356] "Generic (PLEG): container finished" podID="6d3f2e47-48c7-4691-938d-5424e634f2f5" containerID="d82ee008b9bf72c12bac4b4b76a0db899d583d785ac5f7d54fb434326d496247" exitCode=0
Oct 29 22:52:19.924928 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.924929    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp"
Oct 29 22:52:19.925134 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.924971    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp" event={"ID":"6d3f2e47-48c7-4691-938d-5424e634f2f5","Type":"ContainerDied","Data":"d82ee008b9bf72c12bac4b4b76a0db899d583d785ac5f7d54fb434326d496247"}
Oct 29 22:52:19.925134 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.925027    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp" event={"ID":"6d3f2e47-48c7-4691-938d-5424e634f2f5","Type":"ContainerDied","Data":"50bd23e7036ef9a4a3cc81cf5f6b0f0252bea1eb80e04a12153a76e376f6cf5e"}
Oct 29 22:52:19.925134 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.925055    5428 scope.go:122] "RemoveContainer" containerID="d82ee008b9bf72c12bac4b4b76a0db899d583d785ac5f7d54fb434326d496247"
Oct 29 22:52:19.935264 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.935246    5428 scope.go:122] "RemoveContainer" containerID="d82ee008b9bf72c12bac4b4b76a0db899d583d785ac5f7d54fb434326d496247"
Oct 29 22:52:19.935536 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:52:19.935516    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d82ee008b9bf72c12bac4b4b76a0db899d583d785ac5f7d54fb434326d496247\": container with ID starting with d82ee008b9bf72c12bac4b4b76a0db899d583d785ac5f7d54fb434326d496247 not found: ID does not exist" containerID="d82ee008b9bf72c12bac4b4b76a0db899d583d785ac5f7d54fb434326d496247"
Oct 29 22:52:19.935591 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.935547    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82ee008b9bf72c12bac4b4b76a0db899d583d785ac5f7d54fb434326d496247"} err="failed to get container status \"d82ee008b9bf72c12bac4b4b76a0db899d583d785ac5f7d54fb434326d496247\": rpc error: code = NotFound desc = could not find container \"d82ee008b9bf72c12bac4b4b76a0db899d583d785ac5f7d54fb434326d496247\": container with ID starting with d82ee008b9bf72c12bac4b4b76a0db899d583d785ac5f7d54fb434326d496247 not found: ID does not exist"
Oct 29 22:52:19.986160 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.986118    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp"]
Oct 29 22:52:19.988995 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:19.988965    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e406071d-7bc68fffd6-bv5cp"]
Oct 29 22:52:20.600539 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:20.600508    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg"]
Oct 29 22:52:20.709422 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:20.709388    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:20.797295 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:20.797259    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d3f2e47-48c7-4691-938d-5424e634f2f5" path="/var/lib/kubelet/pods/6d3f2e47-48c7-4691-938d-5424e634f2f5/volumes"
Oct 29 22:52:20.851869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:20.851780    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:21.085306 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:21.085263    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:21.787133 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:21.787095    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:21.904960 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:21.904924    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:22.605389 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:22.605346    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:23.421623 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:23.421587    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:23.539230 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:23.539186    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:23.685196 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:23.685111    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:23.805468 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:23.805408    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:23.924567 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:23.924532    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:24.115209 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:24.115170    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:24.239449 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:24.239272    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:24.374564 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:24.374303    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:25.210042 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:25.210015    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:25.601071 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:25.600963    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg" podUID="47d1fe46-10af-40db-9f1c-9c696185d3ea" containerName="payload-pre-processing" containerID="cri-o://ed92fea288cf319b805c850befda3128fb3cd94958509d11f4e3183c73d19fd3" gracePeriod=25
Oct 29 22:52:25.763349 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:25.763303    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:25.871096 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:25.871068    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg"
Oct 29 22:52:25.883738 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:25.883704    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:25.900254 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:25.900223    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj"]
Oct 29 22:52:25.906505 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:25.906440    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e406071d-g9jfj"]
Oct 29 22:52:25.932375 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:25.932334    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d1fe46-10af-40db-9f1c-9c696185d3ea-kube-api-access-r98n4\" (UniqueName: \"kubernetes.io/projected/47d1fe46-10af-40db-9f1c-9c696185d3ea-kube-api-access-r98n4\") pod \"47d1fe46-10af-40db-9f1c-9c696185d3ea\" (UID: \"47d1fe46-10af-40db-9f1c-9c696185d3ea\") "
Oct 29 22:52:25.932561 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:25.932389    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47d1fe46-10af-40db-9f1c-9c696185d3ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d1fe46-10af-40db-9f1c-9c696185d3ea-plugins-config-volume\") pod \"47d1fe46-10af-40db-9f1c-9c696185d3ea\" (UID: \"47d1fe46-10af-40db-9f1c-9c696185d3ea\") "
Oct 29 22:52:25.932954 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:25.932919    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d1fe46-10af-40db-9f1c-9c696185d3ea-plugins-config-volume" pod "47d1fe46-10af-40db-9f1c-9c696185d3ea" (UID: "47d1fe46-10af-40db-9f1c-9c696185d3ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:52:25.934873 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:25.934843    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d1fe46-10af-40db-9f1c-9c696185d3ea-kube-api-access-r98n4" pod "47d1fe46-10af-40db-9f1c-9c696185d3ea" (UID: "47d1fe46-10af-40db-9f1c-9c696185d3ea"). InnerVolumeSpecName "kube-api-access-r98n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:26.031084 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.031046    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"]
Oct 29 22:52:26.032146 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.031928    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs" podUID="887f4003-a75f-4799-8c46-2460a9f89f8e" containerName="istio-proxy" containerID="cri-o://7e85b8a2e5e5c54aabe23ffde9aad982fd57b63cb28865eae3983408e10cf321" gracePeriod=30
Oct 29 22:52:26.033886 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.033826    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r98n4\" (UniqueName: \"kubernetes.io/projected/47d1fe46-10af-40db-9f1c-9c696185d3ea-kube-api-access-r98n4\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:26.033886 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.033856    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47d1fe46-10af-40db-9f1c-9c696185d3ea-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:26.050677 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.049951    5428 generic.go:356] "Generic (PLEG): container finished" podID="47d1fe46-10af-40db-9f1c-9c696185d3ea" containerID="ed92fea288cf319b805c850befda3128fb3cd94958509d11f4e3183c73d19fd3" exitCode=0
Oct 29 22:52:26.050677 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.050007    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg"
Oct 29 22:52:26.050677 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.050060    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg" event={"ID":"47d1fe46-10af-40db-9f1c-9c696185d3ea","Type":"ContainerDied","Data":"ed92fea288cf319b805c850befda3128fb3cd94958509d11f4e3183c73d19fd3"}
Oct 29 22:52:26.050677 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.050124    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg" event={"ID":"47d1fe46-10af-40db-9f1c-9c696185d3ea","Type":"ContainerDied","Data":"80940b1633ea1d76d6750b2623b26dede89be99e7d3ee775273b31f87c226903"}
Oct 29 22:52:26.050677 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.050149    5428 scope.go:122] "RemoveContainer" containerID="ed92fea288cf319b805c850befda3128fb3cd94958509d11f4e3183c73d19fd3"
Oct 29 22:52:26.062462 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.062326    5428 scope.go:122] "RemoveContainer" containerID="ed92fea288cf319b805c850befda3128fb3cd94958509d11f4e3183c73d19fd3"
Oct 29 22:52:26.062754 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:52:26.062701    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed92fea288cf319b805c850befda3128fb3cd94958509d11f4e3183c73d19fd3\": container with ID starting with ed92fea288cf319b805c850befda3128fb3cd94958509d11f4e3183c73d19fd3 not found: ID does not exist" containerID="ed92fea288cf319b805c850befda3128fb3cd94958509d11f4e3183c73d19fd3"
Oct 29 22:52:26.062754 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.062735    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed92fea288cf319b805c850befda3128fb3cd94958509d11f4e3183c73d19fd3"} err="failed to get container status \"ed92fea288cf319b805c850befda3128fb3cd94958509d11f4e3183c73d19fd3\": rpc error: code = NotFound desc = could not find container \"ed92fea288cf319b805c850befda3128fb3cd94958509d11f4e3183c73d19fd3\": container with ID starting with ed92fea288cf319b805c850befda3128fb3cd94958509d11f4e3183c73d19fd3 not found: ID does not exist"
Oct 29 22:52:26.086312 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.086276    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:26.114527 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.114498    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg"]
Oct 29 22:52:26.119245 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.119215    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-865d946f-f46868695-bsglg"]
Oct 29 22:52:26.423033 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.422947    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r"]
Oct 29 22:52:26.536133 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.536095    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:26.653623 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.653585    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:26.778567 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.778478    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47d1fe46-10af-40db-9f1c-9c696185d3ea" path="/var/lib/kubelet/pods/47d1fe46-10af-40db-9f1c-9c696185d3ea/volumes"
Oct 29 22:52:26.778946 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.778929    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbe51a7-678d-4028-872f-02ea1463723e" path="/var/lib/kubelet/pods/bbbe51a7-678d-4028-872f-02ea1463723e/volumes"
Oct 29 22:52:26.879313 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:26.879243    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:27.379240 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:27.379203    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:27.511479 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:27.511437    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:27.919697 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:27.919661    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:28.400276 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:28.400240    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:29.063218 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:29.063164    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:29.070472 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:29.070437    5428 generic.go:356] "Generic (PLEG): container finished" podID="7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" containerID="122b37eccf0ed7fcb6d783ae4cb929f99107edc0f7a1a2e5babe95c59fd99637" exitCode=0
Oct 29 22:52:29.070610 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:29.070507    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp" event={"ID":"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc","Type":"ContainerDied","Data":"122b37eccf0ed7fcb6d783ae4cb929f99107edc0f7a1a2e5babe95c59fd99637"}
Oct 29 22:52:29.070973 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:29.070956    5428 scope.go:122] "RemoveContainer" containerID="122b37eccf0ed7fcb6d783ae4cb929f99107edc0f7a1a2e5babe95c59fd99637"
Oct 29 22:52:29.072407 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:29.072377    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:29.072529 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:29.072462    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:29.072529 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:29.072510    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:29.195567 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:29.195526    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:29.942715 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:29.942679    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"]
Oct 29 22:52:29.943464 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:29.943441    5428 cpu_manager.go:414] "containerMap: removing container" podUID="47d1fe46-10af-40db-9f1c-9c696185d3ea" containerName="payload-pre-processing"
Oct 29 22:52:29.943517 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:29.943467    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d1fe46-10af-40db-9f1c-9c696185d3ea" containerName="payload-pre-processing"
Oct 29 22:52:29.943517 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:29.943493    5428 cpu_manager.go:414] "containerMap: removing container" podUID="6d3f2e47-48c7-4691-938d-5424e634f2f5" containerName="payload-processing"
Oct 29 22:52:29.943517 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:29.943500    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3f2e47-48c7-4691-938d-5424e634f2f5" containerName="payload-processing"
Oct 29 22:52:29.943644 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:29.943633    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3f2e47-48c7-4691-938d-5424e634f2f5" containerName="payload-processing"
Oct 29 22:52:29.943680 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:29.943655    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d1fe46-10af-40db-9f1c-9c696185d3ea" containerName="payload-pre-processing"
Oct 29 22:52:29.947166 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:29.947147    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 22:52:29.980857 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:29.978936    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-32c192-serving-cert\""
Oct 29 22:52:29.991272 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:29.991097    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"]
Oct 29 22:52:30.049544 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.049509    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:30.091977 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.091942    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp" event={"ID":"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc","Type":"ContainerStarted","Data":"aed1b6105d7d4e5f94879c29f415f77abebeaf508b5ea2f7419a6fc1080d94ef"}
Oct 29 22:52:30.102825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.102747    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002743d1-e4e6-43f4-859e-0bcf233b3179-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t\" (UID: \"002743d1-e4e6-43f4-859e-0bcf233b3179\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 22:52:30.103125 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.103098    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002743d1-e4e6-43f4-859e-0bcf233b3179-metrics-certs\") pod \"maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t\" (UID: \"002743d1-e4e6-43f4-859e-0bcf233b3179\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 22:52:30.103192 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.103162    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002743d1-e4e6-43f4-859e-0bcf233b3179-maas-api-tls\") pod \"maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t\" (UID: \"002743d1-e4e6-43f4-859e-0bcf233b3179\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 22:52:30.103333 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.103311    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stbfx\" (UniqueName: \"kubernetes.io/projected/002743d1-e4e6-43f4-859e-0bcf233b3179-kube-api-access-stbfx\") pod \"maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t\" (UID: \"002743d1-e4e6-43f4-859e-0bcf233b3179\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 22:52:30.204512 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.204420    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stbfx\" (UniqueName: \"kubernetes.io/projected/002743d1-e4e6-43f4-859e-0bcf233b3179-kube-api-access-stbfx\") pod \"maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t\" (UID: \"002743d1-e4e6-43f4-859e-0bcf233b3179\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 22:52:30.204678 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.204518    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002743d1-e4e6-43f4-859e-0bcf233b3179-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t\" (UID: \"002743d1-e4e6-43f4-859e-0bcf233b3179\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 22:52:30.204678 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.204608    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002743d1-e4e6-43f4-859e-0bcf233b3179-metrics-certs\") pod \"maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t\" (UID: \"002743d1-e4e6-43f4-859e-0bcf233b3179\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 22:52:30.204678 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.204637    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002743d1-e4e6-43f4-859e-0bcf233b3179-maas-api-tls\") pod \"maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t\" (UID: \"002743d1-e4e6-43f4-859e-0bcf233b3179\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 22:52:30.204911 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:52:30.204840    5428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-serving-cert: secret "maas-api-e2e-worker-w3-32c192-serving-cert" not found
Oct 29 22:52:30.204911 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:52:30.204906    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/002743d1-e4e6-43f4-859e-0bcf233b3179-maas-api-tls podName:002743d1-e4e6-43f4-859e-0bcf233b3179 nodeName:}" failed. No retries permitted until 2026-10-29 22:52:30.704885901 +0000 UTC m=+554.047834624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/002743d1-e4e6-43f4-859e-0bcf233b3179-maas-api-tls") pod "maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t" (UID: "002743d1-e4e6-43f4-859e-0bcf233b3179") : secret "maas-api-e2e-worker-w3-32c192-serving-cert" not found
Oct 29 22:52:30.205318 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.205301    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002743d1-e4e6-43f4-859e-0bcf233b3179-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t\" (UID: \"002743d1-e4e6-43f4-859e-0bcf233b3179\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 22:52:30.207180 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.207153    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002743d1-e4e6-43f4-859e-0bcf233b3179-metrics-certs\") pod \"maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t\" (UID: \"002743d1-e4e6-43f4-859e-0bcf233b3179\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 22:52:30.285505 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.285468    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:30.287007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.286974    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stbfx\" (UniqueName: \"kubernetes.io/projected/002743d1-e4e6-43f4-859e-0bcf233b3179-kube-api-access-stbfx\") pod \"maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t\" (UID: \"002743d1-e4e6-43f4-859e-0bcf233b3179\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 22:52:30.411670 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.411632    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:30.709968 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.709933    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002743d1-e4e6-43f4-859e-0bcf233b3179-maas-api-tls\") pod \"maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t\" (UID: \"002743d1-e4e6-43f4-859e-0bcf233b3179\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 22:52:30.712503 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.712471    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002743d1-e4e6-43f4-859e-0bcf233b3179-maas-api-tls\") pod \"maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t\" (UID: \"002743d1-e4e6-43f4-859e-0bcf233b3179\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 22:52:30.788121 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.788065    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:52:30.788316 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.788144    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp" podUID="7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:52:30.789365 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.789341    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:52:30.871387 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:30.871074    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 22:52:31.219582 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:52:31.219519    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002743d1_e4e6_43f4_859e_0bcf233b3179.slice/crio-4221bc1230c5850bd8f13aa00aa59ab51ebb8e8231308dd27a1eacdf93791b28 WatchSource:0}: Error finding container 4221bc1230c5850bd8f13aa00aa59ab51ebb8e8231308dd27a1eacdf93791b28: Status 404 returned error can't find the container with id 4221bc1230c5850bd8f13aa00aa59ab51ebb8e8231308dd27a1eacdf93791b28
Oct 29 22:52:31.228039 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.228008    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"]
Oct 29 22:52:31.352746 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.352722    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:52:31.423666 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.423594    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r" podUID="909a27ab-47f9-4814-b653-b6c6959ad616" containerName="payload-pre-processing" containerID="cri-o://823057e80cf90960a151d25d86047badcc83fb3c410e24825c0aa7b8ccd2c9fa" gracePeriod=25
Oct 29 22:52:31.523554 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.523519    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-token\" (UniqueName: \"kubernetes.io/projected/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-token\") pod \"887f4003-a75f-4799-8c46-2460a9f89f8e\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") "
Oct 29 22:52:31.523715 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.523582    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/887f4003-a75f-4799-8c46-2460a9f89f8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887f4003-a75f-4799-8c46-2460a9f89f8e-istiod-ca-cert\") pod \"887f4003-a75f-4799-8c46-2460a9f89f8e\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") "
Oct 29 22:52:31.523715 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.523638    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/887f4003-a75f-4799-8c46-2460a9f89f8e-kube-api-access-kwdqt\" (UniqueName: \"kubernetes.io/projected/887f4003-a75f-4799-8c46-2460a9f89f8e-kube-api-access-kwdqt\") pod \"887f4003-a75f-4799-8c46-2460a9f89f8e\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") "
Oct 29 22:52:31.523715 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.523660    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-podinfo\") pod \"887f4003-a75f-4799-8c46-2460a9f89f8e\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") "
Oct 29 22:52:31.523715 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.523697    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-workload-certs\") pod \"887f4003-a75f-4799-8c46-2460a9f89f8e\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") "
Oct 29 22:52:31.523942 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.523736    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-credential-socket\") pod \"887f4003-a75f-4799-8c46-2460a9f89f8e\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") "
Oct 29 22:52:31.523942 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.523920    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-envoy\") pod \"887f4003-a75f-4799-8c46-2460a9f89f8e\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") "
Oct 29 22:52:31.524053 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.523970    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-workload-socket\") pod \"887f4003-a75f-4799-8c46-2460a9f89f8e\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") "
Oct 29 22:52:31.524053 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.524003    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-data\") pod \"887f4003-a75f-4799-8c46-2460a9f89f8e\" (UID: \"887f4003-a75f-4799-8c46-2460a9f89f8e\") "
Oct 29 22:52:31.524053 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.524016    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-workload-certs" pod "887f4003-a75f-4799-8c46-2460a9f89f8e" (UID: "887f4003-a75f-4799-8c46-2460a9f89f8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:52:31.524226 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.524106    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887f4003-a75f-4799-8c46-2460a9f89f8e-istiod-ca-cert" pod "887f4003-a75f-4799-8c46-2460a9f89f8e" (UID: "887f4003-a75f-4799-8c46-2460a9f89f8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:52:31.524282 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.524232    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-workload-socket" pod "887f4003-a75f-4799-8c46-2460a9f89f8e" (UID: "887f4003-a75f-4799-8c46-2460a9f89f8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:52:31.524372 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.524349    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-credential-socket" pod "887f4003-a75f-4799-8c46-2460a9f89f8e" (UID: "887f4003-a75f-4799-8c46-2460a9f89f8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:52:31.524465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.524443    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-data" pod "887f4003-a75f-4799-8c46-2460a9f89f8e" (UID: "887f4003-a75f-4799-8c46-2460a9f89f8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:52:31.524582 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.524558    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:31.524667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.524588    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:31.524667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.524604    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887f4003-a75f-4799-8c46-2460a9f89f8e-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:31.524667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.524618    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:31.524667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.524631    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:31.525938 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.525911    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-token" pod "887f4003-a75f-4799-8c46-2460a9f89f8e" (UID: "887f4003-a75f-4799-8c46-2460a9f89f8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:31.526048 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.525949    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887f4003-a75f-4799-8c46-2460a9f89f8e-kube-api-access-kwdqt" pod "887f4003-a75f-4799-8c46-2460a9f89f8e" (UID: "887f4003-a75f-4799-8c46-2460a9f89f8e"). InnerVolumeSpecName "kube-api-access-kwdqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:31.526048 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.525969    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-envoy" pod "887f4003-a75f-4799-8c46-2460a9f89f8e" (UID: "887f4003-a75f-4799-8c46-2460a9f89f8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:52:31.526048 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.525987    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-podinfo" pod "887f4003-a75f-4799-8c46-2460a9f89f8e" (UID: "887f4003-a75f-4799-8c46-2460a9f89f8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:52:31.589492 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.589258    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:31.626589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.626367    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:31.626589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.626531    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:31.626589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.626545    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwdqt\" (UniqueName: \"kubernetes.io/projected/887f4003-a75f-4799-8c46-2460a9f89f8e-kube-api-access-kwdqt\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:31.626589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.626560    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887f4003-a75f-4799-8c46-2460a9f89f8e-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:31.713408 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.713349    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:31.749014 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.748985    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r"
Oct 29 22:52:31.786962 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.786519    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:52:31.786962 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.786579    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp" podUID="7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:52:31.929319 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.929287    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/909a27ab-47f9-4814-b653-b6c6959ad616-kube-api-access-kqlvz\" (UniqueName: \"kubernetes.io/projected/909a27ab-47f9-4814-b653-b6c6959ad616-kube-api-access-kqlvz\") pod \"909a27ab-47f9-4814-b653-b6c6959ad616\" (UID: \"909a27ab-47f9-4814-b653-b6c6959ad616\") "
Oct 29 22:52:31.929507 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.929473    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/909a27ab-47f9-4814-b653-b6c6959ad616-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909a27ab-47f9-4814-b653-b6c6959ad616-plugins-config-volume\") pod \"909a27ab-47f9-4814-b653-b6c6959ad616\" (UID: \"909a27ab-47f9-4814-b653-b6c6959ad616\") "
Oct 29 22:52:31.929962 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.929929    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909a27ab-47f9-4814-b653-b6c6959ad616-plugins-config-volume" pod "909a27ab-47f9-4814-b653-b6c6959ad616" (UID: "909a27ab-47f9-4814-b653-b6c6959ad616"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:52:31.931600 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:31.931582    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909a27ab-47f9-4814-b653-b6c6959ad616-kube-api-access-kqlvz" pod "909a27ab-47f9-4814-b653-b6c6959ad616" (UID: "909a27ab-47f9-4814-b653-b6c6959ad616"). InnerVolumeSpecName "kube-api-access-kqlvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:32.030531 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.030493    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqlvz\" (UniqueName: \"kubernetes.io/projected/909a27ab-47f9-4814-b653-b6c6959ad616-kube-api-access-kqlvz\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:32.030531 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.030527    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909a27ab-47f9-4814-b653-b6c6959ad616-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:32.114984 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.114949    5428 generic.go:356] "Generic (PLEG): container finished" podID="887f4003-a75f-4799-8c46-2460a9f89f8e" containerID="7e85b8a2e5e5c54aabe23ffde9aad982fd57b63cb28865eae3983408e10cf321" exitCode=0
Oct 29 22:52:32.115183 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.115109    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs" event={"ID":"887f4003-a75f-4799-8c46-2460a9f89f8e","Type":"ContainerDied","Data":"7e85b8a2e5e5c54aabe23ffde9aad982fd57b63cb28865eae3983408e10cf321"}
Oct 29 22:52:32.115250 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.115181    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs" event={"ID":"887f4003-a75f-4799-8c46-2460a9f89f8e","Type":"ContainerDied","Data":"cf807496f7debd786a7a2292c0a75cec96f133f0de9f9c9fde4cd1d65d3a130e"}
Oct 29 22:52:32.115312 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.115238    5428 scope.go:122] "RemoveContainer" containerID="7e85b8a2e5e5c54aabe23ffde9aad982fd57b63cb28865eae3983408e10cf321"
Oct 29 22:52:32.115474 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.115442    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"
Oct 29 22:52:32.119425 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.119405    5428 generic.go:356] "Generic (PLEG): container finished" podID="909a27ab-47f9-4814-b653-b6c6959ad616" containerID="823057e80cf90960a151d25d86047badcc83fb3c410e24825c0aa7b8ccd2c9fa" exitCode=0
Oct 29 22:52:32.119688 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.119468    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r"
Oct 29 22:52:32.120351 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.119490    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r" event={"ID":"909a27ab-47f9-4814-b653-b6c6959ad616","Type":"ContainerDied","Data":"823057e80cf90960a151d25d86047badcc83fb3c410e24825c0aa7b8ccd2c9fa"}
Oct 29 22:52:32.120351 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.120070    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r" event={"ID":"909a27ab-47f9-4814-b653-b6c6959ad616","Type":"ContainerDied","Data":"545ad220747248d65bbc3cbce7b88f59a5df61b34e25a39d5ceb53a196dfdee2"}
Oct 29 22:52:32.122101 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.121977    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t" event={"ID":"002743d1-e4e6-43f4-859e-0bcf233b3179","Type":"ContainerStarted","Data":"ec7eb55b24d664da026bde861e2ba00e3e3cc0a7e2b8e4b30de7938b938913ec"}
Oct 29 22:52:32.122101 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.122020    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t" event={"ID":"002743d1-e4e6-43f4-859e-0bcf233b3179","Type":"ContainerStarted","Data":"4221bc1230c5850bd8f13aa00aa59ab51ebb8e8231308dd27a1eacdf93791b28"}
Oct 29 22:52:32.128858 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.128288    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 22:52:32.137020 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.136928    5428 scope.go:122] "RemoveContainer" containerID="c3d0d38ef97f1c8e80dae32b60b12317e232b9dbc412c47fad567d55f5e72c55"
Oct 29 22:52:32.151858 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.151842    5428 scope.go:122] "RemoveContainer" containerID="7e85b8a2e5e5c54aabe23ffde9aad982fd57b63cb28865eae3983408e10cf321"
Oct 29 22:52:32.152366 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:52:32.152339    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e85b8a2e5e5c54aabe23ffde9aad982fd57b63cb28865eae3983408e10cf321\": container with ID starting with 7e85b8a2e5e5c54aabe23ffde9aad982fd57b63cb28865eae3983408e10cf321 not found: ID does not exist" containerID="7e85b8a2e5e5c54aabe23ffde9aad982fd57b63cb28865eae3983408e10cf321"
Oct 29 22:52:32.152434 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.152380    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e85b8a2e5e5c54aabe23ffde9aad982fd57b63cb28865eae3983408e10cf321"} err="failed to get container status \"7e85b8a2e5e5c54aabe23ffde9aad982fd57b63cb28865eae3983408e10cf321\": rpc error: code = NotFound desc = could not find container \"7e85b8a2e5e5c54aabe23ffde9aad982fd57b63cb28865eae3983408e10cf321\": container with ID starting with 7e85b8a2e5e5c54aabe23ffde9aad982fd57b63cb28865eae3983408e10cf321 not found: ID does not exist"
Oct 29 22:52:32.152434 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.152425    5428 scope.go:122] "RemoveContainer" containerID="c3d0d38ef97f1c8e80dae32b60b12317e232b9dbc412c47fad567d55f5e72c55"
Oct 29 22:52:32.152976 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:52:32.152955    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d0d38ef97f1c8e80dae32b60b12317e232b9dbc412c47fad567d55f5e72c55\": container with ID starting with c3d0d38ef97f1c8e80dae32b60b12317e232b9dbc412c47fad567d55f5e72c55 not found: ID does not exist" containerID="c3d0d38ef97f1c8e80dae32b60b12317e232b9dbc412c47fad567d55f5e72c55"
Oct 29 22:52:32.153036 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.153015    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d0d38ef97f1c8e80dae32b60b12317e232b9dbc412c47fad567d55f5e72c55"} err="failed to get container status \"c3d0d38ef97f1c8e80dae32b60b12317e232b9dbc412c47fad567d55f5e72c55\": rpc error: code = NotFound desc = could not find container \"c3d0d38ef97f1c8e80dae32b60b12317e232b9dbc412c47fad567d55f5e72c55\": container with ID starting with c3d0d38ef97f1c8e80dae32b60b12317e232b9dbc412c47fad567d55f5e72c55 not found: ID does not exist"
Oct 29 22:52:32.153088 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.153039    5428 scope.go:122] "RemoveContainer" containerID="823057e80cf90960a151d25d86047badcc83fb3c410e24825c0aa7b8ccd2c9fa"
Oct 29 22:52:32.168824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.159173    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:32.168824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.164634    5428 scope.go:122] "RemoveContainer" containerID="823057e80cf90960a151d25d86047badcc83fb3c410e24825c0aa7b8ccd2c9fa"
Oct 29 22:52:32.170304 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:52:32.170230    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823057e80cf90960a151d25d86047badcc83fb3c410e24825c0aa7b8ccd2c9fa\": container with ID starting with 823057e80cf90960a151d25d86047badcc83fb3c410e24825c0aa7b8ccd2c9fa not found: ID does not exist" containerID="823057e80cf90960a151d25d86047badcc83fb3c410e24825c0aa7b8ccd2c9fa"
Oct 29 22:52:32.170304 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.170267    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823057e80cf90960a151d25d86047badcc83fb3c410e24825c0aa7b8ccd2c9fa"} err="failed to get container status \"823057e80cf90960a151d25d86047badcc83fb3c410e24825c0aa7b8ccd2c9fa\": rpc error: code = NotFound desc = could not find container \"823057e80cf90960a151d25d86047badcc83fb3c410e24825c0aa7b8ccd2c9fa\": container with ID starting with 823057e80cf90960a151d25d86047badcc83fb3c410e24825c0aa7b8ccd2c9fa not found: ID does not exist"
Oct 29 22:52:32.191317 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.191250    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"]
Oct 29 22:52:32.194704 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.194675    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e406071d-openshift-default-55545c5bd5-gxghs"]
Oct 29 22:52:32.234285 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.234059    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t" podStartSLOduration=2.8179073470000002 podStartE2EDuration="3.234047517s" totalImagesPullingTime="416.14017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:32.232828996 +0000 UTC m=+555.575777739" watchObservedRunningTime="2026-10-29 22:52:32.234047517 +0000 UTC m=+555.576996262"
Oct 29 22:52:32.284528 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.284491    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r"]
Oct 29 22:52:32.285711 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.285671    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8042816e-6f58588b75-6zp9r"]
Oct 29 22:52:32.286517 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.286481    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:32.392922 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.391336    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5"]
Oct 29 22:52:32.394471 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.394449    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865d946f-l6bl5"]
Oct 29 22:52:32.406765 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.406741    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:32.532939 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.532546    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"]
Oct 29 22:52:32.533438 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.533184    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn" podUID="fd186355-c594-4b2b-b716-21ced7841757" containerName="istio-proxy" containerID="cri-o://e6297090865be69d90f35aa6c8ed2107c2763f1b707043510109558e9740ef0a" gracePeriod=30
Oct 29 22:52:32.591312 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.591190    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:32.592496 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.592466    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:52:32.788745 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.788644    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:52:32.788745 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.788708    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp" podUID="7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:52:32.790536 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.790507    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="887f4003-a75f-4799-8c46-2460a9f89f8e" path="/var/lib/kubelet/pods/887f4003-a75f-4799-8c46-2460a9f89f8e/volumes"
Oct 29 22:52:32.791010 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.790990    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="909a27ab-47f9-4814-b653-b6c6959ad616" path="/var/lib/kubelet/pods/909a27ab-47f9-4814-b653-b6c6959ad616/volumes"
Oct 29 22:52:32.791434 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.791416    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfac23f3-7a49-4e44-8123-cf9a0286174d" path="/var/lib/kubelet/pods/cfac23f3-7a49-4e44-8123-cf9a0286174d/volumes"
Oct 29 22:52:32.822246 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.822212    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:32.931940 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:32.931906    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:33.051227 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:33.051143    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:33.383864 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:33.383669    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:33.610915 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:33.610796    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:33.741315 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:33.741221    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:33.787067 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:33.787029    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:52:33.787244 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:33.787085    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp" podUID="7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:52:33.889370 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:33.889335    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:34.023402 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:34.023320    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:34.626628 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:34.626540    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:34.784504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:34.784465    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:34.787259 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:34.787231    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:52:34.787402 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:34.787302    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp" podUID="7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:52:35.426685 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:35.426652    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:35.791229 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:35.791150    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:52:35.792188 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:35.792166    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 22:52:35.858264 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:35.858227    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"]
Oct 29 22:52:35.859112 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:35.859074    5428 cpu_manager.go:414] "containerMap: removing container" podUID="887f4003-a75f-4799-8c46-2460a9f89f8e" containerName="istio-proxy"
Oct 29 22:52:35.859112 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:35.859111    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="887f4003-a75f-4799-8c46-2460a9f89f8e" containerName="istio-proxy"
Oct 29 22:52:35.859291 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:35.859125    5428 cpu_manager.go:414] "containerMap: removing container" podUID="909a27ab-47f9-4814-b653-b6c6959ad616" containerName="payload-pre-processing"
Oct 29 22:52:35.859291 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:35.859133    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="909a27ab-47f9-4814-b653-b6c6959ad616" containerName="payload-pre-processing"
Oct 29 22:52:35.859380 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:35.859354    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="887f4003-a75f-4799-8c46-2460a9f89f8e" containerName="istio-proxy"
Oct 29 22:52:35.859380 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:35.859373    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="887f4003-a75f-4799-8c46-2460a9f89f8e" containerName="istio-proxy"
Oct 29 22:52:35.859475 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:35.859392    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="909a27ab-47f9-4814-b653-b6c6959ad616" containerName="payload-pre-processing"
Oct 29 22:52:35.866663 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:35.866637    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:35.881914 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:35.881890    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ed8218-openshift-default-dockercfg-x28tf\""
Oct 29 22:52:35.887755 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:35.887731    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"]
Oct 29 22:52:35.965055 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:35.965010    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:36.016410 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.016376    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.016555 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.016445    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.016607 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.016571    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.016607 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.016603    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.016667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.016626    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.016733 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.016717    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqvbd\" (UniqueName: \"kubernetes.io/projected/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-kube-api-access-wqvbd\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.016768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.016746    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.016812 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.016770    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.016958 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.016930    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.118498 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.118462    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.118682 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.118504    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.118682 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.118525    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.118682 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.118562    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqvbd\" (UniqueName: \"kubernetes.io/projected/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-kube-api-access-wqvbd\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.118682 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.118588    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.118682 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.118612    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.118682 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.118665    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.119001 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.118720    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.119001 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.118826    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.119165 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.119139    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.119234 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.119199    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.119289 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.119250    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.119408 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.119391    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.119642 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.119620    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.121352 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.121321    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.121827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.121788    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.140412 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.140383    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqvbd\" (UniqueName: \"kubernetes.io/projected/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-kube-api-access-wqvbd\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.141168 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.141143    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.191601 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.191559    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:36.191601 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.191587    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:36.602239 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.600880    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"]
Oct 29 22:52:36.608214 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.608173    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:36.608419 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.608394    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:36.608635 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.608610    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:36.783865 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.783561    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:36.906760 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:36.906726    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:37.137879 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.137833    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:37.172898 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.168303    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd" event={"ID":"ae1fd856-9cb0-4d65-bc25-49aefe12e84a","Type":"ContainerStarted","Data":"4dfd90cc1c3f82298cffd572d81380e54c81d3b035b41168b1a0c350092c9659"}
Oct 29 22:52:37.172898 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.168377    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd" event={"ID":"ae1fd856-9cb0-4d65-bc25-49aefe12e84a","Type":"ContainerStarted","Data":"157639e6794c0cf21cd5a66791e767d72d0e770b1eccc63b244f75acb687dd73"}
Oct 29 22:52:37.192185 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.192147    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:37.199227 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.199196    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:37.237608 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.237546    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd" podStartSLOduration=2.237531723 podStartE2EDuration="2.237531723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:37.225529102 +0000 UTC m=+560.568477848" watchObservedRunningTime="2026-10-29 22:52:37.237531723 +0000 UTC m=+560.580480458"
Oct 29 22:52:37.252782 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.252728    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:37.847733 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.847707    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:52:37.945434 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.945401    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-istio-data\") pod \"fd186355-c594-4b2b-b716-21ced7841757\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") "
Oct 29 22:52:37.945434 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.945435    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-credential-socket\") pod \"fd186355-c594-4b2b-b716-21ced7841757\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") "
Oct 29 22:52:37.945975 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.945465    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd186355-c594-4b2b-b716-21ced7841757-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd186355-c594-4b2b-b716-21ced7841757-istio-podinfo\") pod \"fd186355-c594-4b2b-b716-21ced7841757\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") "
Oct 29 22:52:37.945975 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.945485    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd186355-c594-4b2b-b716-21ced7841757-istio-token\" (UniqueName: \"kubernetes.io/projected/fd186355-c594-4b2b-b716-21ced7841757-istio-token\") pod \"fd186355-c594-4b2b-b716-21ced7841757\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") "
Oct 29 22:52:37.945975 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.945543    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd186355-c594-4b2b-b716-21ced7841757-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd186355-c594-4b2b-b716-21ced7841757-istiod-ca-cert\") pod \"fd186355-c594-4b2b-b716-21ced7841757\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") "
Oct 29 22:52:37.945975 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.945560    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-workload-socket\") pod \"fd186355-c594-4b2b-b716-21ced7841757\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") "
Oct 29 22:52:37.945975 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.945650    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-workload-certs\") pod \"fd186355-c594-4b2b-b716-21ced7841757\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") "
Oct 29 22:52:37.945975 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.945713    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-istio-envoy\") pod \"fd186355-c594-4b2b-b716-21ced7841757\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") "
Oct 29 22:52:37.945975 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.945754    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd186355-c594-4b2b-b716-21ced7841757-kube-api-access-255tx\" (UniqueName: \"kubernetes.io/projected/fd186355-c594-4b2b-b716-21ced7841757-kube-api-access-255tx\") pod \"fd186355-c594-4b2b-b716-21ced7841757\" (UID: \"fd186355-c594-4b2b-b716-21ced7841757\") "
Oct 29 22:52:37.945975 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.945776    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-istio-data" pod "fd186355-c594-4b2b-b716-21ced7841757" (UID: "fd186355-c594-4b2b-b716-21ced7841757"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:52:37.945975 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.945966    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-workload-certs" pod "fd186355-c594-4b2b-b716-21ced7841757" (UID: "fd186355-c594-4b2b-b716-21ced7841757"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:52:37.946412 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.945976    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-workload-socket" pod "fd186355-c594-4b2b-b716-21ced7841757" (UID: "fd186355-c594-4b2b-b716-21ced7841757"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:52:37.946412 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.946265    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:37.946412 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.946288    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:37.946412 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.946302    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:37.946412 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.946359    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-credential-socket" pod "fd186355-c594-4b2b-b716-21ced7841757" (UID: "fd186355-c594-4b2b-b716-21ced7841757"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:52:37.946412 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.946367    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd186355-c594-4b2b-b716-21ced7841757-istiod-ca-cert" pod "fd186355-c594-4b2b-b716-21ced7841757" (UID: "fd186355-c594-4b2b-b716-21ced7841757"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:52:37.948072 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.948040    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd186355-c594-4b2b-b716-21ced7841757-kube-api-access-255tx" pod "fd186355-c594-4b2b-b716-21ced7841757" (UID: "fd186355-c594-4b2b-b716-21ced7841757"). InnerVolumeSpecName "kube-api-access-255tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:37.948305 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.948281    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd186355-c594-4b2b-b716-21ced7841757-istio-podinfo" pod "fd186355-c594-4b2b-b716-21ced7841757" (UID: "fd186355-c594-4b2b-b716-21ced7841757"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:52:37.948438 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.948413    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-istio-envoy" pod "fd186355-c594-4b2b-b716-21ced7841757" (UID: "fd186355-c594-4b2b-b716-21ced7841757"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:52:37.948502 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:37.948463    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd186355-c594-4b2b-b716-21ced7841757-istio-token" pod "fd186355-c594-4b2b-b716-21ced7841757" (UID: "fd186355-c594-4b2b-b716-21ced7841757"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:38.047521 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.047482    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd186355-c594-4b2b-b716-21ced7841757-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:38.047521 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.047517    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:38.047521 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.047527    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-255tx\" (UniqueName: \"kubernetes.io/projected/fd186355-c594-4b2b-b716-21ced7841757-kube-api-access-255tx\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:38.047823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.047536    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd186355-c594-4b2b-b716-21ced7841757-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:38.047823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.047546    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd186355-c594-4b2b-b716-21ced7841757-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:38.047823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.047556    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd186355-c594-4b2b-b716-21ced7841757-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:52:38.136993 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.136966    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 22:52:38.194689 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.194586    5428 generic.go:356] "Generic (PLEG): container finished" podID="fd186355-c594-4b2b-b716-21ced7841757" containerID="e6297090865be69d90f35aa6c8ed2107c2763f1b707043510109558e9740ef0a" exitCode=0
Oct 29 22:52:38.195834 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.195786    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"
Oct 29 22:52:38.198258 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.198221    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn" event={"ID":"fd186355-c594-4b2b-b716-21ced7841757","Type":"ContainerDied","Data":"e6297090865be69d90f35aa6c8ed2107c2763f1b707043510109558e9740ef0a"}
Oct 29 22:52:38.198376 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.198299    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:38.198376 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.198327    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn" event={"ID":"fd186355-c594-4b2b-b716-21ced7841757","Type":"ContainerDied","Data":"2af4bda0aa4ef654ea16a97cc6cfa3872db4c1646fb6391826bbb873baee65d7"}
Oct 29 22:52:38.198376 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.198356    5428 scope.go:122] "RemoveContainer" containerID="e6297090865be69d90f35aa6c8ed2107c2763f1b707043510109558e9740ef0a"
Oct 29 22:52:38.199107 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.199006    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:52:38.213084 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.213068    5428 scope.go:122] "RemoveContainer" containerID="d0cb063f35742e4738e0e3fb88629419b88866f7719cdf487f10ec82e54f5e2d"
Oct 29 22:52:38.223452 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.223431    5428 scope.go:122] "RemoveContainer" containerID="e6297090865be69d90f35aa6c8ed2107c2763f1b707043510109558e9740ef0a"
Oct 29 22:52:38.223762 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:52:38.223742    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6297090865be69d90f35aa6c8ed2107c2763f1b707043510109558e9740ef0a\": container with ID starting with e6297090865be69d90f35aa6c8ed2107c2763f1b707043510109558e9740ef0a not found: ID does not exist" containerID="e6297090865be69d90f35aa6c8ed2107c2763f1b707043510109558e9740ef0a"
Oct 29 22:52:38.223872 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.223773    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6297090865be69d90f35aa6c8ed2107c2763f1b707043510109558e9740ef0a"} err="failed to get container status \"e6297090865be69d90f35aa6c8ed2107c2763f1b707043510109558e9740ef0a\": rpc error: code = NotFound desc = could not find container \"e6297090865be69d90f35aa6c8ed2107c2763f1b707043510109558e9740ef0a\": container with ID starting with e6297090865be69d90f35aa6c8ed2107c2763f1b707043510109558e9740ef0a not found: ID does not exist"
Oct 29 22:52:38.223872 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.223827    5428 scope.go:122] "RemoveContainer" containerID="d0cb063f35742e4738e0e3fb88629419b88866f7719cdf487f10ec82e54f5e2d"
Oct 29 22:52:38.224112 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:52:38.224093    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0cb063f35742e4738e0e3fb88629419b88866f7719cdf487f10ec82e54f5e2d\": container with ID starting with d0cb063f35742e4738e0e3fb88629419b88866f7719cdf487f10ec82e54f5e2d not found: ID does not exist" containerID="d0cb063f35742e4738e0e3fb88629419b88866f7719cdf487f10ec82e54f5e2d"
Oct 29 22:52:38.224175 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.224120    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0cb063f35742e4738e0e3fb88629419b88866f7719cdf487f10ec82e54f5e2d"} err="failed to get container status \"d0cb063f35742e4738e0e3fb88629419b88866f7719cdf487f10ec82e54f5e2d\": rpc error: code = NotFound desc = could not find container \"d0cb063f35742e4738e0e3fb88629419b88866f7719cdf487f10ec82e54f5e2d\": container with ID starting with d0cb063f35742e4738e0e3fb88629419b88866f7719cdf487f10ec82e54f5e2d not found: ID does not exist"
Oct 29 22:52:38.249605 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.249572    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"]
Oct 29 22:52:38.255334 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.255304    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-865d946f-openshift-default-f68cc5c65-nqtbn"]
Oct 29 22:52:38.789471 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.789436    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd186355-c594-4b2b-b716-21ced7841757" path="/var/lib/kubelet/pods/fd186355-c594-4b2b-b716-21ced7841757/volumes"
Oct 29 22:52:38.908451 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:38.908417    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:39.031575 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.031535    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:39.033950 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.033924    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc"]
Oct 29 22:52:39.034427 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.034410    5428 cpu_manager.go:414] "containerMap: removing container" podUID="887f4003-a75f-4799-8c46-2460a9f89f8e" containerName="istio-proxy"
Oct 29 22:52:39.034468 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.034429    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="887f4003-a75f-4799-8c46-2460a9f89f8e" containerName="istio-proxy"
Oct 29 22:52:39.034468 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.034445    5428 cpu_manager.go:414] "containerMap: removing container" podUID="fd186355-c594-4b2b-b716-21ced7841757" containerName="istio-proxy"
Oct 29 22:52:39.034468 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.034450    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd186355-c594-4b2b-b716-21ced7841757" containerName="istio-proxy"
Oct 29 22:52:39.034468 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.034464    5428 cpu_manager.go:414] "containerMap: removing container" podUID="fd186355-c594-4b2b-b716-21ced7841757" containerName="istio-proxy"
Oct 29 22:52:39.034468 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.034469    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd186355-c594-4b2b-b716-21ced7841757" containerName="istio-proxy"
Oct 29 22:52:39.034614 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.034539    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd186355-c594-4b2b-b716-21ced7841757" containerName="istio-proxy"
Oct 29 22:52:39.039364 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.039343    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc"
Oct 29 22:52:39.052532 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.052506    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-7aeaad-dockercfg-4l84k\""
Oct 29 22:52:39.083836 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.082846    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc"]
Oct 29 22:52:39.083836 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.082996    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-7aeaad\""
Oct 29 22:52:39.089823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.087915    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2rjm\" (UniqueName: \"kubernetes.io/projected/8b10d05c-ccd8-4125-819e-19e71a233aa9-kube-api-access-p2rjm\") pod \"payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc\" (UID: \"8b10d05c-ccd8-4125-819e-19e71a233aa9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc"
Oct 29 22:52:39.089823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.088004    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b10d05c-ccd8-4125-819e-19e71a233aa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc\" (UID: \"8b10d05c-ccd8-4125-819e-19e71a233aa9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc"
Oct 29 22:52:39.201873 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.195897    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2rjm\" (UniqueName: \"kubernetes.io/projected/8b10d05c-ccd8-4125-819e-19e71a233aa9-kube-api-access-p2rjm\") pod \"payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc\" (UID: \"8b10d05c-ccd8-4125-819e-19e71a233aa9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc"
Oct 29 22:52:39.201873 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.195995    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b10d05c-ccd8-4125-819e-19e71a233aa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc\" (UID: \"8b10d05c-ccd8-4125-819e-19e71a233aa9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc"
Oct 29 22:52:39.201873 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.196272    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:39.201873 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.197012    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b10d05c-ccd8-4125-819e-19e71a233aa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc\" (UID: \"8b10d05c-ccd8-4125-819e-19e71a233aa9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc"
Oct 29 22:52:39.242996 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.242868    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2rjm\" (UniqueName: \"kubernetes.io/projected/8b10d05c-ccd8-4125-819e-19e71a233aa9-kube-api-access-p2rjm\") pod \"payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc\" (UID: \"8b10d05c-ccd8-4125-819e-19e71a233aa9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc"
Oct 29 22:52:39.353865 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.353832    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc"
Oct 29 22:52:39.687085 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:39.687059    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc"]
Oct 29 22:52:39.690107 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:52:39.690065    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b10d05c_ccd8_4125_819e_19e71a233aa9.slice/crio-e6ac909b22e0b4b69282c59d38026c31e94f38065da3f0a53eae7d5934dc5358 WatchSource:0}: Error finding container e6ac909b22e0b4b69282c59d38026c31e94f38065da3f0a53eae7d5934dc5358: Status 404 returned error can't find the container with id e6ac909b22e0b4b69282c59d38026c31e94f38065da3f0a53eae7d5934dc5358
Oct 29 22:52:40.246571 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:40.246531    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc" event={"ID":"8b10d05c-ccd8-4125-819e-19e71a233aa9","Type":"ContainerStarted","Data":"92d5122dcc6d92b3449f39354b078177d5e9d1ccf559028ef57df6b6a24844dc"}
Oct 29 22:52:40.247067 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:40.246594    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc" event={"ID":"8b10d05c-ccd8-4125-819e-19e71a233aa9","Type":"ContainerStarted","Data":"e6ac909b22e0b4b69282c59d38026c31e94f38065da3f0a53eae7d5934dc5358"}
Oct 29 22:52:40.247067 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:40.246646    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc"
Oct 29 22:52:40.288674 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:40.288610    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc" podStartSLOduration=1.288592132 podStartE2EDuration="1.288592132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:40.286772069 +0000 UTC m=+563.629720814" watchObservedRunningTime="2026-10-29 22:52:40.288592132 +0000 UTC m=+563.631540890"
Oct 29 22:52:40.413538 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:40.413485    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:40.559923 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:40.559662    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:40.950295 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:40.950264    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:42.094106 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.093674    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:42.195939 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.195904    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7"]
Oct 29 22:52:42.196658 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.196631    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd186355-c594-4b2b-b716-21ced7841757" containerName="istio-proxy"
Oct 29 22:52:42.200271 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.200250    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7"
Oct 29 22:52:42.223687 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.223658    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-ed8218-dockercfg-n44pb\""
Oct 29 22:52:42.223867 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.223730    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-ed8218\""
Oct 29 22:52:42.227061 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.227034    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7"]
Oct 29 22:52:42.228446 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.228400    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:42.231066 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.231042    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f08d29e-2d26-454f-9667-8c3b99f6855d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7\" (UID: \"6f08d29e-2d26-454f-9667-8c3b99f6855d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7"
Oct 29 22:52:42.231154 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.231124    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx45b\" (UniqueName: \"kubernetes.io/projected/6f08d29e-2d26-454f-9667-8c3b99f6855d-kube-api-access-kx45b\") pod \"payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7\" (UID: \"6f08d29e-2d26-454f-9667-8c3b99f6855d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7"
Oct 29 22:52:42.252885 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.250049    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"]
Oct 29 22:52:42.254325 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.254299    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"
Oct 29 22:52:42.284337 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.284239    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"]
Oct 29 22:52:42.336826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.336288    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f08d29e-2d26-454f-9667-8c3b99f6855d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7\" (UID: \"6f08d29e-2d26-454f-9667-8c3b99f6855d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7"
Oct 29 22:52:42.336826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.336354    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5180d7a3-b01d-4639-9421-f802b9155fa2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff\" (UID: \"5180d7a3-b01d-4639-9421-f802b9155fa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"
Oct 29 22:52:42.336826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.336387    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx45b\" (UniqueName: \"kubernetes.io/projected/6f08d29e-2d26-454f-9667-8c3b99f6855d-kube-api-access-kx45b\") pod \"payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7\" (UID: \"6f08d29e-2d26-454f-9667-8c3b99f6855d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7"
Oct 29 22:52:42.336826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.336470    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7kp8\" (UniqueName: \"kubernetes.io/projected/5180d7a3-b01d-4639-9421-f802b9155fa2-kube-api-access-s7kp8\") pod \"payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff\" (UID: \"5180d7a3-b01d-4639-9421-f802b9155fa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"
Oct 29 22:52:42.347762 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.343077    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f08d29e-2d26-454f-9667-8c3b99f6855d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7\" (UID: \"6f08d29e-2d26-454f-9667-8c3b99f6855d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7"
Oct 29 22:52:42.390335 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.390304    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx45b\" (UniqueName: \"kubernetes.io/projected/6f08d29e-2d26-454f-9667-8c3b99f6855d-kube-api-access-kx45b\") pod \"payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7\" (UID: \"6f08d29e-2d26-454f-9667-8c3b99f6855d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7"
Oct 29 22:52:42.442539 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.442489    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7kp8\" (UniqueName: \"kubernetes.io/projected/5180d7a3-b01d-4639-9421-f802b9155fa2-kube-api-access-s7kp8\") pod \"payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff\" (UID: \"5180d7a3-b01d-4639-9421-f802b9155fa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"
Oct 29 22:52:42.442718 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.442673    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5180d7a3-b01d-4639-9421-f802b9155fa2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff\" (UID: \"5180d7a3-b01d-4639-9421-f802b9155fa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"
Oct 29 22:52:42.443529 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.443503    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5180d7a3-b01d-4639-9421-f802b9155fa2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff\" (UID: \"5180d7a3-b01d-4639-9421-f802b9155fa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"
Oct 29 22:52:42.466762 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.466724    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7kp8\" (UniqueName: \"kubernetes.io/projected/5180d7a3-b01d-4639-9421-f802b9155fa2-kube-api-access-s7kp8\") pod \"payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff\" (UID: \"5180d7a3-b01d-4639-9421-f802b9155fa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"
Oct 29 22:52:42.518926 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.518891    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7"
Oct 29 22:52:42.569996 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.569963    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"
Oct 29 22:52:42.840498 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.840463    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7"]
Oct 29 22:52:42.842377 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:52:42.842348    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f08d29e_2d26_454f_9667_8c3b99f6855d.slice/crio-6c9562c2bcc5ccfe78913edda26750c5cbe56e550e30d02993c6c4cd753bccea WatchSource:0}: Error finding container 6c9562c2bcc5ccfe78913edda26750c5cbe56e550e30d02993c6c4cd753bccea: Status 404 returned error can't find the container with id 6c9562c2bcc5ccfe78913edda26750c5cbe56e550e30d02993c6c4cd753bccea
Oct 29 22:52:42.949926 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.949894    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:42.961095 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:42.961069    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"]
Oct 29 22:52:42.963361 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:52:42.963330    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5180d7a3_b01d_4639_9421_f802b9155fa2.slice/crio-ab864367a05556056a1e1b87aede485a4a3222edf823b27a5c19c3abb045e186 WatchSource:0}: Error finding container ab864367a05556056a1e1b87aede485a4a3222edf823b27a5c19c3abb045e186: Status 404 returned error can't find the container with id ab864367a05556056a1e1b87aede485a4a3222edf823b27a5c19c3abb045e186
Oct 29 22:52:43.070143 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:43.070112    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:43.188668 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:43.188582    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:43.271880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:43.271837    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7" event={"ID":"6f08d29e-2d26-454f-9667-8c3b99f6855d","Type":"ContainerStarted","Data":"c6827f0beee74dd6d3e013e2db3b060391f2f3cad34a2c6d366a1ee5fb5cf5f0"}
Oct 29 22:52:43.272053 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:43.271907    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7" event={"ID":"6f08d29e-2d26-454f-9667-8c3b99f6855d","Type":"ContainerStarted","Data":"6c9562c2bcc5ccfe78913edda26750c5cbe56e550e30d02993c6c4cd753bccea"}
Oct 29 22:52:43.272130 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:43.272045    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7"
Oct 29 22:52:43.274059 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:43.273911    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff" event={"ID":"5180d7a3-b01d-4639-9421-f802b9155fa2","Type":"ContainerStarted","Data":"69b36197811fa0e3274a486b874a3a5fddd57f70425cacedb8ad2ca0f04b7906"}
Oct 29 22:52:43.274059 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:43.273956    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff" event={"ID":"5180d7a3-b01d-4639-9421-f802b9155fa2","Type":"ContainerStarted","Data":"ab864367a05556056a1e1b87aede485a4a3222edf823b27a5c19c3abb045e186"}
Oct 29 22:52:43.274059 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:43.274007    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"
Oct 29 22:52:43.314458 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:43.314401    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7" podStartSLOduration=1.3143865909999999 podStartE2EDuration="1.314386591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:43.312044848 +0000 UTC m=+566.654993622" watchObservedRunningTime="2026-10-29 22:52:43.314386591 +0000 UTC m=+566.657335362"
Oct 29 22:52:43.355158 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:43.355095    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff" podStartSLOduration=1.355074905 podStartE2EDuration="1.355074905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:43.352178452 +0000 UTC m=+566.695127222" watchObservedRunningTime="2026-10-29 22:52:43.355074905 +0000 UTC m=+566.698023655"
Oct 29 22:52:43.644942 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:43.644911    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:44.266833 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:44.266770    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:44.401651 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:44.401617    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:44.519102 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:44.519035    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:44.895998 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:44.895927    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:46.296046 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:46.296013    5428 generic.go:356] "Generic (PLEG): container finished" podID="751014a5-e82e-4403-83b7-f2e648514119" containerID="80c7dcab7d14ea41d1fe5c5dcb6307b187bd8377b64b6d99b3c72e48e4a1b7d9" exitCode=0
Oct 29 22:52:46.296474 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:46.296088    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh" event={"ID":"751014a5-e82e-4403-83b7-f2e648514119","Type":"ContainerDied","Data":"80c7dcab7d14ea41d1fe5c5dcb6307b187bd8377b64b6d99b3c72e48e4a1b7d9"}
Oct 29 22:52:46.296521 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:46.296491    5428 scope.go:122] "RemoveContainer" containerID="80c7dcab7d14ea41d1fe5c5dcb6307b187bd8377b64b6d99b3c72e48e4a1b7d9"
Oct 29 22:52:46.297919 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:46.297891    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:46.298003 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:46.297977    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:46.298040 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:46.298010    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:52:46.607091 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:46.607002    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:46.906985 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:46.906944    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:47.342155 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:47.342041    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh" event={"ID":"751014a5-e82e-4403-83b7-f2e648514119","Type":"ContainerStarted","Data":"b166f7ea3195f011c8695a6d64b7dff6262d2c43cd621e6370964a3dbb90b78e"}
Oct 29 22:52:47.352158 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:47.352134    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:47.358615 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:47.358587    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:52:47.359090 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:47.359054    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh" podUID="751014a5-e82e-4403-83b7-f2e648514119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 22:52:47.442821 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:52:47.434532    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:52:48.281156 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.281122    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"]
Oct 29 22:52:48.295970 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.295863    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 22:52:48.301925 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.301877    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"]
Oct 29 22:52:48.309405 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.309376    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-7aeaad-serving-cert\""
Oct 29 22:52:48.347183 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.347147    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:52:48.347574 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.347207    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh" podUID="751014a5-e82e-4403-83b7-f2e648514119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 22:52:48.350078 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.350052    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf20849b-2d40-42ca-82fa-a91198bb012b-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f\" (UID: \"cf20849b-2d40-42ca-82fa-a91198bb012b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 22:52:48.350228 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.350112    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dsnx\" (UniqueName: \"kubernetes.io/projected/cf20849b-2d40-42ca-82fa-a91198bb012b-kube-api-access-8dsnx\") pod \"maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f\" (UID: \"cf20849b-2d40-42ca-82fa-a91198bb012b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 22:52:48.350385 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.350365    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf20849b-2d40-42ca-82fa-a91198bb012b-metrics-certs\") pod \"maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f\" (UID: \"cf20849b-2d40-42ca-82fa-a91198bb012b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 22:52:48.350453 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.350407    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf20849b-2d40-42ca-82fa-a91198bb012b-maas-api-tls\") pod \"maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f\" (UID: \"cf20849b-2d40-42ca-82fa-a91198bb012b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 22:52:48.395693 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.395646    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:48.451200 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.451159    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf20849b-2d40-42ca-82fa-a91198bb012b-metrics-certs\") pod \"maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f\" (UID: \"cf20849b-2d40-42ca-82fa-a91198bb012b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 22:52:48.451394 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.451212    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf20849b-2d40-42ca-82fa-a91198bb012b-maas-api-tls\") pod \"maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f\" (UID: \"cf20849b-2d40-42ca-82fa-a91198bb012b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 22:52:48.451394 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.451247    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf20849b-2d40-42ca-82fa-a91198bb012b-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f\" (UID: \"cf20849b-2d40-42ca-82fa-a91198bb012b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 22:52:48.451394 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.451290    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dsnx\" (UniqueName: \"kubernetes.io/projected/cf20849b-2d40-42ca-82fa-a91198bb012b-kube-api-access-8dsnx\") pod \"maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f\" (UID: \"cf20849b-2d40-42ca-82fa-a91198bb012b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 22:52:48.451572 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:52:48.451466    5428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-serving-cert: secret "maas-api-e2e-worker-w2-7aeaad-serving-cert" not found
Oct 29 22:52:48.451572 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:52:48.451532    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf20849b-2d40-42ca-82fa-a91198bb012b-maas-api-tls podName:cf20849b-2d40-42ca-82fa-a91198bb012b nodeName:}" failed. No retries permitted until 2026-10-29 22:52:48.951511223 +0000 UTC m=+572.294459950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf20849b-2d40-42ca-82fa-a91198bb012b-maas-api-tls") pod "maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f" (UID: "cf20849b-2d40-42ca-82fa-a91198bb012b") : secret "maas-api-e2e-worker-w2-7aeaad-serving-cert" not found
Oct 29 22:52:48.452059 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.452038    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf20849b-2d40-42ca-82fa-a91198bb012b-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f\" (UID: \"cf20849b-2d40-42ca-82fa-a91198bb012b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 22:52:48.454415 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.454391    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf20849b-2d40-42ca-82fa-a91198bb012b-metrics-certs\") pod \"maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f\" (UID: \"cf20849b-2d40-42ca-82fa-a91198bb012b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 22:52:48.510646 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.510613    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dsnx\" (UniqueName: \"kubernetes.io/projected/cf20849b-2d40-42ca-82fa-a91198bb012b-kube-api-access-8dsnx\") pod \"maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f\" (UID: \"cf20849b-2d40-42ca-82fa-a91198bb012b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 22:52:48.515726 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.515658    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:48.726651 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.726617    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:48.957241 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.957196    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf20849b-2d40-42ca-82fa-a91198bb012b-maas-api-tls\") pod \"maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f\" (UID: \"cf20849b-2d40-42ca-82fa-a91198bb012b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 22:52:48.979867 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:48.979388    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf20849b-2d40-42ca-82fa-a91198bb012b-maas-api-tls\") pod \"maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f\" (UID: \"cf20849b-2d40-42ca-82fa-a91198bb012b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 22:52:49.213257 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:49.213224    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 22:52:49.348216 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:49.348182    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:52:49.348568 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:49.348243    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh" podUID="751014a5-e82e-4403-83b7-f2e648514119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 22:52:49.553569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:49.553483    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"]
Oct 29 22:52:49.556787 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:52:49.556761    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf20849b_2d40_42ca_82fa_a91198bb012b.slice/crio-358f98366ccac8d23fa58fc88145b171448320315069b01c81e7c5b3e3e8ce3d WatchSource:0}: Error finding container 358f98366ccac8d23fa58fc88145b171448320315069b01c81e7c5b3e3e8ce3d: Status 404 returned error can't find the container with id 358f98366ccac8d23fa58fc88145b171448320315069b01c81e7c5b3e3e8ce3d
Oct 29 22:52:50.346450 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:50.346373    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:52:50.346450 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:50.346428    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh" podUID="751014a5-e82e-4403-83b7-f2e648514119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 22:52:50.385139 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:50.385100    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f" event={"ID":"cf20849b-2d40-42ca-82fa-a91198bb012b","Type":"ContainerStarted","Data":"fb5c1d8b8db505ed68fcd767ed628c29626a31b55d520fcadf9efaddeb3b1ee4"}
Oct 29 22:52:50.385650 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:50.385170    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f" event={"ID":"cf20849b-2d40-42ca-82fa-a91198bb012b","Type":"ContainerStarted","Data":"358f98366ccac8d23fa58fc88145b171448320315069b01c81e7c5b3e3e8ce3d"}
Oct 29 22:52:50.385650 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:50.385338    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 22:52:50.400660 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:50.400632    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:50.424736 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:50.424684    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f" podStartSLOduration=2.084810847 podStartE2EDuration="2.424671169s" totalImagesPullingTime="339.860322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:50.42310528 +0000 UTC m=+573.766054026" watchObservedRunningTime="2026-10-29 22:52:50.424671169 +0000 UTC m=+573.767619911"
Oct 29 22:52:51.254692 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:51.254442    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc"
Oct 29 22:52:51.347510 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:51.347465    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:52:51.347697 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:51.347533    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh" podUID="751014a5-e82e-4403-83b7-f2e648514119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 22:52:52.230534 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:52.230501    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:52.347160 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:52.347121    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:52:52.347369 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:52.347201    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh" podUID="751014a5-e82e-4403-83b7-f2e648514119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 22:52:52.353431 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:52.353396    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:52.667620 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:52.667582    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:53.350956 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:53.350927    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:53.351372 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:53.351195    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:53.351785 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:53.351767    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 22:52:54.285741 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:54.285707    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7"
Oct 29 22:52:54.286354 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:54.286324    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"
Oct 29 22:52:54.399153 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:54.399116    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:54.975133 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:54.975099    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:56.271129 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:56.271090    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:56.397454 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:56.397431    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 22:52:57.604031 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:57.603989    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:57.718706 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:57.718674    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:58.040794 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:58.040749    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:58.309909 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:58.309819    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:58.493812 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:58.493771    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:58.676153 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:52:58.676119    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:00.016107 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:00.016068    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:01.887909 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:01.887324    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:02.018230 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:02.018195    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:02.513471 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:02.513435    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:02.651706 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:02.651671    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:02.767985 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:02.767905    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:04.335849 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:04.335794    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:06.149475 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:06.149441    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:07.405986 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.405954    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:07.585465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.585429    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"]
Oct 29 22:53:07.589637 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.589614    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.604980 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.604955    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"]
Oct 29 22:53:07.611708 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.611675    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e57216-openshift-default-dockercfg-jh4hn\""
Oct 29 22:53:07.631316 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.631292    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.631433 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.631330    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.631480 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.631465    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2hx5\" (UniqueName: \"kubernetes.io/projected/75a16c52-71c1-4da1-bc73-4e6fea6184c8-kube-api-access-p2hx5\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.631520 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.631497    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.631683 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.631633    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.631835 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.631699    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.631835 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.631733    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.631835 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.631750    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.632028 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.631855    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.733786 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.733699    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.733786 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.733733    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.733786 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.733766    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.734083 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.733871    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.734083 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.733919    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.734083 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.733979    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.734083 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.734014    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.734312 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.734111    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hx5\" (UniqueName: \"kubernetes.io/projected/75a16c52-71c1-4da1-bc73-4e6fea6184c8-kube-api-access-p2hx5\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.734312 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.734155    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.734312 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.734157    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.734312 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.734275    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.734499 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.734397    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.734499 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.734413    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.734786 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.734766    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.736314 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.736291    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.736473 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.736456    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.755526 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.755497    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.755655 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.755569    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2hx5\" (UniqueName: \"kubernetes.io/projected/75a16c52-71c1-4da1-bc73-4e6fea6184c8-kube-api-access-p2hx5\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:07.903597 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:07.903561    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:08.041530 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:08.041485    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:08.191828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:08.184503    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:08.266213 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:08.266108    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"]
Oct 29 22:53:08.274125 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:53:08.274084    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a16c52_71c1_4da1_bc73_4e6fea6184c8.slice/crio-88040f1faadc2f2e3641a81aa8a26c9d58cbe7cc56977289ba4733d333a9d1ea WatchSource:0}: Error finding container 88040f1faadc2f2e3641a81aa8a26c9d58cbe7cc56977289ba4733d333a9d1ea: Status 404 returned error can't find the container with id 88040f1faadc2f2e3641a81aa8a26c9d58cbe7cc56977289ba4733d333a9d1ea
Oct 29 22:53:08.276972 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:08.276936    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:53:08.277059 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:08.277020    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:53:08.277103 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:08.277058    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:53:08.304723 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:08.304686    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:08.551662 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:08.551560    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x" event={"ID":"75a16c52-71c1-4da1-bc73-4e6fea6184c8","Type":"ContainerStarted","Data":"f5af76081a2c09f4a2b14c5fda368876ea76b3b70c9098a3bd55340fa656711e"}
Oct 29 22:53:08.551662 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:08.551633    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x" event={"ID":"75a16c52-71c1-4da1-bc73-4e6fea6184c8","Type":"ContainerStarted","Data":"88040f1faadc2f2e3641a81aa8a26c9d58cbe7cc56977289ba4733d333a9d1ea"}
Oct 29 22:53:08.605676 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:08.605612    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x" podStartSLOduration=1.605591032 podStartE2EDuration="1.605591032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:08.602981543 +0000 UTC m=+591.945930288" watchObservedRunningTime="2026-10-29 22:53:08.605591032 +0000 UTC m=+591.948539778"
Oct 29 22:53:08.904222 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:08.904175    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:09.906172 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:09.906138    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:53:09.906684 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:09.906190    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x" podUID="75a16c52-71c1-4da1-bc73-4e6fea6184c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:53:09.959303 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:09.959199    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:10.287143 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:10.287006    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:10.413683 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:10.413623    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:10.904052 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:10.903998    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:53:10.904252 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:10.904067    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x" podUID="75a16c52-71c1-4da1-bc73-4e6fea6184c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:53:11.757188 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:11.757148    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:11.904142 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:11.904094    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:53:11.904333 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:11.904183    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x" podUID="75a16c52-71c1-4da1-bc73-4e6fea6184c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:53:12.904427 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:12.904389    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:53:12.906031 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:12.904457    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x" podUID="75a16c52-71c1-4da1-bc73-4e6fea6184c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:53:13.471826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:13.469909    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:13.637756 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:13.637721    5428 generic.go:356] "Generic (PLEG): container finished" podID="ae1fd856-9cb0-4d65-bc25-49aefe12e84a" containerID="4dfd90cc1c3f82298cffd572d81380e54c81d3b035b41168b1a0c350092c9659" exitCode=0
Oct 29 22:53:13.637971 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:13.637794    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd" event={"ID":"ae1fd856-9cb0-4d65-bc25-49aefe12e84a","Type":"ContainerDied","Data":"4dfd90cc1c3f82298cffd572d81380e54c81d3b035b41168b1a0c350092c9659"}
Oct 29 22:53:13.638228 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:13.638210    5428 scope.go:122] "RemoveContainer" containerID="4dfd90cc1c3f82298cffd572d81380e54c81d3b035b41168b1a0c350092c9659"
Oct 29 22:53:13.639534 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:13.639509    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:53:13.639590 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:13.639581    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:53:13.639624 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:13.639613    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:53:13.904701 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:13.904359    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:53:13.904701 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:13.904418    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x" podUID="75a16c52-71c1-4da1-bc73-4e6fea6184c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:53:14.645742 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:14.645706    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd" event={"ID":"ae1fd856-9cb0-4d65-bc25-49aefe12e84a","Type":"ContainerStarted","Data":"c783e485126991a6cb5dc6acbebe6b1e208c9377dd1b110e7e90a1cf8862e358"}
Oct 29 22:53:15.096377 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:15.096270    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:15.197153 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:15.197121    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:53:15.197357 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:15.197246    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:53:15.197357 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:15.197293    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd" podUID="ae1fd856-9cb0-4d65-bc25-49aefe12e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:53:15.212441 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:15.212407    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:15.747941 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:15.747905    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:15.872871 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:15.872835    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:15.944998 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:15.944964    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:53:15.945158 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:15.945051    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x" podUID="75a16c52-71c1-4da1-bc73-4e6fea6184c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:16.021515 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:16.021417    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:16.022332 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:16.022259    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:16.023205 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:16.023183    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 22:53:16.100017 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:16.099961    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:16.194256 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:16.194217    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:53:16.194992 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:16.194951    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd" podUID="ae1fd856-9cb0-4d65-bc25-49aefe12e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:53:16.350609 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:16.350469    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:16.989178 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:16.989130    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:17.203870 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:17.203001    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:53:17.203870 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:17.203070    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd" podUID="ae1fd856-9cb0-4d65-bc25-49aefe12e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:53:17.452846 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:53:17.450579    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:53:18.192531 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:18.192497    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:53:18.192708 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:18.192547    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd" podUID="ae1fd856-9cb0-4d65-bc25-49aefe12e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:53:18.743532 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:18.743496    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:19.192303 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:19.192265    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:53:19.192488 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:19.192322    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd" podUID="ae1fd856-9cb0-4d65-bc25-49aefe12e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:53:20.197099 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:20.197066    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:53:20.197584 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:20.197471    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:53:20.198229 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:20.198207    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 22:53:20.493769 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:20.493678    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:22.170470 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:22.170434    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:23.866121 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:23.866076    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:23.988952 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:23.988915    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:25.561784 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:25.561748    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:26.110761 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:26.110727    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:27.284711 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:27.284664    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:28.968070 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:28.968031    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:30.593450 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:30.593390    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:30.965916 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:30.965878    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:32.111428 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:32.111395    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:32.273474 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:32.273441    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:34.158010 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:34.157938    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:35.676275 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:35.676240    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:37.291290 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:37.291254    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:38.833264 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:38.833229    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:38.953643 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:38.953608    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:40.612048 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:40.612009    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:42.382143 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:42.382106    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:43.965698 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:43.965663    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:44.688421 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.688382    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"]
Oct 29 22:53:44.692590 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.692566    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:44.711366 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.711341    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"]
Oct 29 22:53:44.718243 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.718221    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-32c192\"/\"premium-w3-32c192-kserve-self-signed-certs\""
Oct 29 22:53:44.718470 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.718448    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-32c192\"/\"default-dockercfg-rl87m\""
Oct 29 22:53:44.718544 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.718468    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-32c192\"/\"openshift-service-ca.crt\""
Oct 29 22:53:44.718544 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.718484    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-32c192\"/\"kube-root-ca.crt\""
Oct 29 22:53:44.761737 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.761700    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:44.813333 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.813305    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-dshm\") pod \"premium-w3-32c192-kserve-5c5d6b698f-5gqnr\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") " pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:44.813456 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.813362    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-tmp-dir\") pod \"premium-w3-32c192-kserve-5c5d6b698f-5gqnr\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") " pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:44.813456 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.813441    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsdxs\" (UniqueName: \"kubernetes.io/projected/9d18c7a0-ce12-453d-88cf-7f8de30371ac-kube-api-access-fsdxs\") pod \"premium-w3-32c192-kserve-5c5d6b698f-5gqnr\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") " pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:44.813547 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.813530    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-home\") pod \"premium-w3-32c192-kserve-5c5d6b698f-5gqnr\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") " pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:44.813589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.813575    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-model-cache\") pod \"premium-w3-32c192-kserve-5c5d6b698f-5gqnr\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") " pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:44.813711 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.813695    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d18c7a0-ce12-453d-88cf-7f8de30371ac-tls-certs\") pod \"premium-w3-32c192-kserve-5c5d6b698f-5gqnr\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") " pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:44.914543 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.914505    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-home\") pod \"premium-w3-32c192-kserve-5c5d6b698f-5gqnr\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") " pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:44.914746 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.914602    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-model-cache\") pod \"premium-w3-32c192-kserve-5c5d6b698f-5gqnr\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") " pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:44.914746 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.914703    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d18c7a0-ce12-453d-88cf-7f8de30371ac-tls-certs\") pod \"premium-w3-32c192-kserve-5c5d6b698f-5gqnr\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") " pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:44.914964 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.914782    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-dshm\") pod \"premium-w3-32c192-kserve-5c5d6b698f-5gqnr\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") " pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:44.914964 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.914881    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-tmp-dir\") pod \"premium-w3-32c192-kserve-5c5d6b698f-5gqnr\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") " pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:44.915086 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.914962    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsdxs\" (UniqueName: \"kubernetes.io/projected/9d18c7a0-ce12-453d-88cf-7f8de30371ac-kube-api-access-fsdxs\") pod \"premium-w3-32c192-kserve-5c5d6b698f-5gqnr\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") " pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:44.915399 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.915372    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-home\") pod \"premium-w3-32c192-kserve-5c5d6b698f-5gqnr\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") " pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:44.915529 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.915456    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-model-cache\") pod \"premium-w3-32c192-kserve-5c5d6b698f-5gqnr\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") " pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:44.917825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.917758    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-tmp-dir\") pod \"premium-w3-32c192-kserve-5c5d6b698f-5gqnr\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") " pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:44.918199 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.918178    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-dshm\") pod \"premium-w3-32c192-kserve-5c5d6b698f-5gqnr\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") " pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:44.918938 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.918918    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d18c7a0-ce12-453d-88cf-7f8de30371ac-tls-certs\") pod \"premium-w3-32c192-kserve-5c5d6b698f-5gqnr\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") " pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:44.936395 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:44.936371    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsdxs\" (UniqueName: \"kubernetes.io/projected/9d18c7a0-ce12-453d-88cf-7f8de30371ac-kube-api-access-fsdxs\") pod \"premium-w3-32c192-kserve-5c5d6b698f-5gqnr\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") " pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:45.006457 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:45.006366    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:45.310594 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:45.310568    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"]
Oct 29 22:53:45.313349 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:53:45.313318    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d18c7a0_ce12_453d_88cf_7f8de30371ac.slice/crio-ab7f05ad2666c0dde2666308e23dad1a06ccf2fe041d4288efb0d31c3bfb4f9d WatchSource:0}: Error finding container ab7f05ad2666c0dde2666308e23dad1a06ccf2fe041d4288efb0d31c3bfb4f9d: Status 404 returned error can't find the container with id ab7f05ad2666c0dde2666308e23dad1a06ccf2fe041d4288efb0d31c3bfb4f9d
Oct 29 22:53:45.578824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:45.578716    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:45.873558 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:45.873520    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" event={"ID":"9d18c7a0-ce12-453d-88cf-7f8de30371ac","Type":"ContainerStarted","Data":"61dde9e7557330a23f0882a0c7b222aec7cf5a8538d053a85aa64435afc892e2"}
Oct 29 22:53:45.873744 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:45.873580    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" event={"ID":"9d18c7a0-ce12-453d-88cf-7f8de30371ac","Type":"ContainerStarted","Data":"ab7f05ad2666c0dde2666308e23dad1a06ccf2fe041d4288efb0d31c3bfb4f9d"}
Oct 29 22:53:45.912717 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:45.912641    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podStartSLOduration=1.912621413 podStartE2EDuration="1.912621413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:45.911101415 +0000 UTC m=+629.254050160" watchObservedRunningTime="2026-10-29 22:53:45.912621413 +0000 UTC m=+629.255570159"
Oct 29 22:53:46.007465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:46.007437    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:47.296040 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:47.296001    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:47.371730 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:53:47.371688    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:53:48.882707 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:48.882650    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:49.879744 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:49.879690    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:50.582478 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:50.582439    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:52.249256 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:52.249202    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:53.782221 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:53.782181    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:55.007341 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:55.007297    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:55.020917 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:55.020891    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:55.657247 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:55.657217    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:55.953786 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:55.953707    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:53:57.375828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:57.375768    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:59.037758 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:53:59.037723    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:00.740950 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:00.740912    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:02.363205 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:02.363174    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:02.864125 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:02.864089    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"]
Oct 29 22:54:02.868226 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:02.868203    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:02.883647 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:02.883611    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ed8218\"/\"kube-root-ca.crt\""
Oct 29 22:54:02.887130 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:02.887109    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"]
Oct 29 22:54:02.894071 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:02.894044    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ed8218\"/\"default-dockercfg-bdgrh\""
Oct 29 22:54:02.894178 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:02.894135    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ed8218\"/\"premium-w4-ed8218-kserve-self-signed-certs\""
Oct 29 22:54:02.919359 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:02.919335    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:02.927650 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:02.927627    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvpr4\" (UniqueName: \"kubernetes.io/projected/90966e36-5c24-4642-97a3-dbe4825b6a10-kube-api-access-qvpr4\") pod \"premium-w4-ed8218-kserve-766c8d6bc6-t26ct\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") " pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:02.927743 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:02.927678    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-dshm\") pod \"premium-w4-ed8218-kserve-766c8d6bc6-t26ct\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") " pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:02.927787 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:02.927762    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-model-cache\") pod \"premium-w4-ed8218-kserve-766c8d6bc6-t26ct\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") " pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:02.927850 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:02.927795    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90966e36-5c24-4642-97a3-dbe4825b6a10-tls-certs\") pod \"premium-w4-ed8218-kserve-766c8d6bc6-t26ct\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") " pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:02.927947 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:02.927932    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-home\") pod \"premium-w4-ed8218-kserve-766c8d6bc6-t26ct\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") " pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:02.927996 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:02.927985    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-tmp-dir\") pod \"premium-w4-ed8218-kserve-766c8d6bc6-t26ct\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") " pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:02.983655 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:02.983628    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ed8218\"/\"openshift-service-ca.crt\""
Oct 29 22:54:03.029597 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:03.029562    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90966e36-5c24-4642-97a3-dbe4825b6a10-tls-certs\") pod \"premium-w4-ed8218-kserve-766c8d6bc6-t26ct\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") " pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:03.029751 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:03.029636    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-home\") pod \"premium-w4-ed8218-kserve-766c8d6bc6-t26ct\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") " pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:03.029751 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:03.029680    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-tmp-dir\") pod \"premium-w4-ed8218-kserve-766c8d6bc6-t26ct\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") " pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:03.029751 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:03.029725    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvpr4\" (UniqueName: \"kubernetes.io/projected/90966e36-5c24-4642-97a3-dbe4825b6a10-kube-api-access-qvpr4\") pod \"premium-w4-ed8218-kserve-766c8d6bc6-t26ct\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") " pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:03.029902 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:03.029759    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-dshm\") pod \"premium-w4-ed8218-kserve-766c8d6bc6-t26ct\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") " pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:03.029902 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:03.029786    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-model-cache\") pod \"premium-w4-ed8218-kserve-766c8d6bc6-t26ct\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") " pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:03.030179 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:03.030141    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-home\") pod \"premium-w4-ed8218-kserve-766c8d6bc6-t26ct\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") " pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:03.030179 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:03.030159    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-tmp-dir\") pod \"premium-w4-ed8218-kserve-766c8d6bc6-t26ct\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") " pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:03.030354 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:03.030221    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-model-cache\") pod \"premium-w4-ed8218-kserve-766c8d6bc6-t26ct\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") " pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:03.031970 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:03.031942    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-dshm\") pod \"premium-w4-ed8218-kserve-766c8d6bc6-t26ct\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") " pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:03.032280 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:03.032258    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90966e36-5c24-4642-97a3-dbe4825b6a10-tls-certs\") pod \"premium-w4-ed8218-kserve-766c8d6bc6-t26ct\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") " pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:03.082852 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:03.082317    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvpr4\" (UniqueName: \"kubernetes.io/projected/90966e36-5c24-4642-97a3-dbe4825b6a10-kube-api-access-qvpr4\") pod \"premium-w4-ed8218-kserve-766c8d6bc6-t26ct\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") " pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:03.181516 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:03.181428    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:03.483089 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:03.483058    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"]
Oct 29 22:54:03.485886 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:54:03.485859    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90966e36_5c24_4642_97a3_dbe4825b6a10.slice/crio-68c89279d17b8e4fef9f38e97c5f2bc387b7ad8655d81ab764291123a948cb27 WatchSource:0}: Error finding container 68c89279d17b8e4fef9f38e97c5f2bc387b7ad8655d81ab764291123a948cb27: Status 404 returned error can't find the container with id 68c89279d17b8e4fef9f38e97c5f2bc387b7ad8655d81ab764291123a948cb27
Oct 29 22:54:03.981783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:03.981748    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:04.005857 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:04.005821    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" event={"ID":"90966e36-5c24-4642-97a3-dbe4825b6a10","Type":"ContainerStarted","Data":"9ce48f686f5b438e9ee6617c2ec850607c5b0eecb59f9e4c3876e8770d1dd43e"}
Oct 29 22:54:04.006019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:04.005881    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" event={"ID":"90966e36-5c24-4642-97a3-dbe4825b6a10","Type":"ContainerStarted","Data":"68c89279d17b8e4fef9f38e97c5f2bc387b7ad8655d81ab764291123a948cb27"}
Oct 29 22:54:04.053784 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:04.053726    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podStartSLOduration=2.053712243 podStartE2EDuration="2.053712243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:04.050197764 +0000 UTC m=+647.393146509" watchObservedRunningTime="2026-10-29 22:54:04.053712243 +0000 UTC m=+647.396660987"
Oct 29 22:54:04.182435 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:04.182363    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:05.725667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:05.725630    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:07.429977 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:07.429937    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:09.108921 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:09.108885    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:10.774937 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:10.774897    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:12.372208 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:12.372170    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:13.182338 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:13.182304    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:13.194960 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:13.194932    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:14.083910 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:14.083882    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:54:14.090456 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:14.090424    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:14.884909 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:14.884861    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"]
Oct 29 22:54:14.892482 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:14.892450    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:14.911309 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:14.911281    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"]
Oct 29 22:54:14.916390 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:14.916365    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7aeaad\"/\"default-dockercfg-c28wt\""
Oct 29 22:54:14.916517 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:14.916414    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7aeaad\"/\"premium-w2-7aeaad-kserve-self-signed-certs\""
Oct 29 22:54:14.922858 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:14.922835    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7aeaad\"/\"openshift-service-ca.crt\""
Oct 29 22:54:14.922954 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:14.922926    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7aeaad\"/\"kube-root-ca.crt\""
Oct 29 22:54:14.957687 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:14.957657    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:14.969714 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:14.969686    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-home\") pod \"premium-w2-7aeaad-kserve-7864b989bf-bpzq6\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") " pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:14.969957 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:14.969924    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv72h\" (UniqueName: \"kubernetes.io/projected/8df2f41d-d961-4041-9125-685f4d3a375f-kube-api-access-wv72h\") pod \"premium-w2-7aeaad-kserve-7864b989bf-bpzq6\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") " pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:14.970185 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:14.970166    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df2f41d-d961-4041-9125-685f4d3a375f-tls-certs\") pod \"premium-w2-7aeaad-kserve-7864b989bf-bpzq6\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") " pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:14.970401 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:14.970383    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-tmp-dir\") pod \"premium-w2-7aeaad-kserve-7864b989bf-bpzq6\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") " pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:14.970630 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:14.970613    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-dshm\") pod \"premium-w2-7aeaad-kserve-7864b989bf-bpzq6\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") " pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:14.970824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:14.970782    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-model-cache\") pod \"premium-w2-7aeaad-kserve-7864b989bf-bpzq6\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") " pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:15.096124 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:15.096086    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-tmp-dir\") pod \"premium-w2-7aeaad-kserve-7864b989bf-bpzq6\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") " pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:15.097614 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:15.097588    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-dshm\") pod \"premium-w2-7aeaad-kserve-7864b989bf-bpzq6\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") " pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:15.097853 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:15.097835    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-model-cache\") pod \"premium-w2-7aeaad-kserve-7864b989bf-bpzq6\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") " pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:15.098015 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:15.098000    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-home\") pod \"premium-w2-7aeaad-kserve-7864b989bf-bpzq6\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") " pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:15.098133 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:15.098119    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv72h\" (UniqueName: \"kubernetes.io/projected/8df2f41d-d961-4041-9125-685f4d3a375f-kube-api-access-wv72h\") pod \"premium-w2-7aeaad-kserve-7864b989bf-bpzq6\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") " pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:15.098285 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:15.098272    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df2f41d-d961-4041-9125-685f4d3a375f-tls-certs\") pod \"premium-w2-7aeaad-kserve-7864b989bf-bpzq6\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") " pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:15.112766 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:15.103979    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df2f41d-d961-4041-9125-685f4d3a375f-tls-certs\") pod \"premium-w2-7aeaad-kserve-7864b989bf-bpzq6\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") " pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:15.112766 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:15.104261    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-model-cache\") pod \"premium-w2-7aeaad-kserve-7864b989bf-bpzq6\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") " pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:15.112766 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:15.104621    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-home\") pod \"premium-w2-7aeaad-kserve-7864b989bf-bpzq6\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") " pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:15.112766 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:15.097408    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-tmp-dir\") pod \"premium-w2-7aeaad-kserve-7864b989bf-bpzq6\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") " pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:15.119209 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:15.119148    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-dshm\") pod \"premium-w2-7aeaad-kserve-7864b989bf-bpzq6\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") " pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:15.131052 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:15.131026    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv72h\" (UniqueName: \"kubernetes.io/projected/8df2f41d-d961-4041-9125-685f4d3a375f-kube-api-access-wv72h\") pod \"premium-w2-7aeaad-kserve-7864b989bf-bpzq6\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") " pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:15.213297 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:15.213211    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:15.520663 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:15.520638    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"]
Oct 29 22:54:15.523390 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:54:15.523358    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8df2f41d_d961_4041_9125_685f4d3a375f.slice/crio-c79d92f093061d63a23ec65e46f282e5f48d2a3c5911a7cfaf0b9607ae35ba0e WatchSource:0}: Error finding container c79d92f093061d63a23ec65e46f282e5f48d2a3c5911a7cfaf0b9607ae35ba0e: Status 404 returned error can't find the container with id c79d92f093061d63a23ec65e46f282e5f48d2a3c5911a7cfaf0b9607ae35ba0e
Oct 29 22:54:15.769407 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:15.769375    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:16.123719 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:16.123681    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" event={"ID":"8df2f41d-d961-4041-9125-685f4d3a375f","Type":"ContainerStarted","Data":"fdd204d364f4e8d88e235dd92f474a38c7a879ab279a99af4b9474bf3207c230"}
Oct 29 22:54:16.124116 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:16.123733    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" event={"ID":"8df2f41d-d961-4041-9125-685f4d3a375f","Type":"ContainerStarted","Data":"c79d92f093061d63a23ec65e46f282e5f48d2a3c5911a7cfaf0b9607ae35ba0e"}
Oct 29 22:54:16.164657 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:16.164594    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podStartSLOduration=2.164573041 podStartE2EDuration="2.164573041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:16.161676292 +0000 UTC m=+659.504625036" watchObservedRunningTime="2026-10-29 22:54:16.164573041 +0000 UTC m=+659.507521788"
Oct 29 22:54:16.214300 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:16.214268    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:17.367415 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:54:17.367373    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:54:17.471758 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:17.471707    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:19.195823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:19.195541    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:20.958237 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:20.958198    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:22.656604 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:22.656570    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:23.964483 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:23.964453    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:24.355571 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:24.355478    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:25.214264 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:25.214226    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:25.227083 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:25.227057    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:26.066572 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:26.066534    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:26.224876 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:26.224846    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:54:27.469273 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:27.469240    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:27.782600 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:27.782509    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:29.486628 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:29.486590    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:31.074214 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:31.074170    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:32.781596 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:32.781564    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:34.682430 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:34.682388    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:36.427298 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:36.427259    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:38.027538 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:38.027497    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:39.637409 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:39.637371    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:40.752420 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:40.752382    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:40.861135 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:40.861090    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:41.232253 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:41.232216    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:42.486647 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:42.486609    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:42.937683 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:42.937644    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:44.295438 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:44.295406    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:44.616051 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:44.616014    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:46.369671 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:46.369636    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:47.367662 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:54:47.367622    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:54:48.114056 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:48.114022    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:49.671070 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:49.671036    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:50.006065 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:50.005987    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:51.696748 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:51.696711    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:53.271872 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:53.271841    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:54.878179 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:54.878143    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:56.554892 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:56.554859    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:57.779859 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.779822    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"]
Oct 29 22:54:57.784962 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.784938    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.797837 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.797812    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-1ee571-openshift-default-dockercfg-bmxwc\""
Oct 29 22:54:57.813030 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.813004    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"]
Oct 29 22:54:57.897174 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.897133    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.897459 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.897435    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8c4f\" (UniqueName: \"kubernetes.io/projected/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-kube-api-access-n8c4f\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.897574 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.897554    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.897653 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.897595    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.897710 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.897698    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.897779 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.897761    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.897882 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.897837    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.897944 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.897887    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.897996 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.897981    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.998787 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.998751    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.998992 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.998813    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8c4f\" (UniqueName: \"kubernetes.io/projected/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-kube-api-access-n8c4f\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.998992 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.998849    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.998992 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.998876    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.998992 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.998946    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.999268 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.999232    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.999268 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.999253    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.999392 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.999268    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.999392 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.999293    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.999392 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.999344    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.999557 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.999436    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.999557 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.999517    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:57.999695 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:57.999602    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:58.000260 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:58.000233    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:58.001688 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:58.001664    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:58.002307 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:58.002283    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:58.020516 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:58.020494    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8c4f\" (UniqueName: \"kubernetes.io/projected/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-kube-api-access-n8c4f\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:58.020938 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:58.020917    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:58.100917 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:58.100829    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:54:58.291137 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:58.291103    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:58.435158 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:58.435128    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"]
Oct 29 22:54:58.440836 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:54:58.439010    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7dec0d7_a62e_4d35_975b_cbe9bb9470e4.slice/crio-4fad2e1d7205b2cd2a9abeace95be22d4201f29426a0ad7db9ef93c6be9b86e8 WatchSource:0}: Error finding container 4fad2e1d7205b2cd2a9abeace95be22d4201f29426a0ad7db9ef93c6be9b86e8: Status 404 returned error can't find the container with id 4fad2e1d7205b2cd2a9abeace95be22d4201f29426a0ad7db9ef93c6be9b86e8
Oct 29 22:54:58.443290 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:58.443251    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:54:58.443395 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:58.443332    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:54:58.443395 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:58.443363    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:54:59.447870 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:59.446632    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm" event={"ID":"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4","Type":"ContainerStarted","Data":"65a3081b7e301eadacc7f5de4f0b8136e2b3c84f5be1d8895eea76ec71a8b3c1"}
Oct 29 22:54:59.447870 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:59.446694    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm" event={"ID":"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4","Type":"ContainerStarted","Data":"4fad2e1d7205b2cd2a9abeace95be22d4201f29426a0ad7db9ef93c6be9b86e8"}
Oct 29 22:54:59.512231 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:59.512167    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm" podStartSLOduration=2.512151838 podStartE2EDuration="2.512151838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:59.509990775 +0000 UTC m=+702.852939520" watchObservedRunningTime="2026-10-29 22:54:59.512151838 +0000 UTC m=+702.855100583"
Oct 29 22:54:59.908450 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:54:59.908416    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:00.102042 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:00.102008    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:55:00.103782 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:00.103754    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:55:00.103953 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:00.103825    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm" podUID="c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 29 22:55:01.102135 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:01.102101    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:55:01.102511 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:01.102152    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm" podUID="c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 29 22:55:01.319981 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:01.319948    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:01.638019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:01.637984    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:02.101270 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:02.101188    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:55:02.101270 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:02.101250    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm" podUID="c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 29 22:55:03.102038 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:03.102002    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:55:03.102420 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:03.102075    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm" podUID="c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 29 22:55:03.287759 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:03.287711    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:03.820956 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:03.820919    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"]
Oct 29 22:55:03.824838 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:03.824796    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:03.847938 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:03.847908    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-32c192\"/\"distinct-w3-32c192-kserve-self-signed-certs\""
Oct 29 22:55:03.852970 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:03.852945    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"]
Oct 29 22:55:03.885724 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:03.885684    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:03.997574 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:03.997536    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znc5d\" (UniqueName: \"kubernetes.io/projected/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-kube-api-access-znc5d\") pod \"distinct-w3-32c192-kserve-6bc48b64dd-vkk6g\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:03.997744 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:03.997605    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-tmp-dir\") pod \"distinct-w3-32c192-kserve-6bc48b64dd-vkk6g\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:03.997744 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:03.997696    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-tls-certs\") pod \"distinct-w3-32c192-kserve-6bc48b64dd-vkk6g\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:03.997829 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:03.997743    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-home\") pod \"distinct-w3-32c192-kserve-6bc48b64dd-vkk6g\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:03.997887 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:03.997872    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-dshm\") pod \"distinct-w3-32c192-kserve-6bc48b64dd-vkk6g\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:03.997971 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:03.997960    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-model-cache\") pod \"distinct-w3-32c192-kserve-6bc48b64dd-vkk6g\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:04.100724 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:04.099849    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-model-cache\") pod \"distinct-w3-32c192-kserve-6bc48b64dd-vkk6g\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:04.100724 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:04.099958    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znc5d\" (UniqueName: \"kubernetes.io/projected/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-kube-api-access-znc5d\") pod \"distinct-w3-32c192-kserve-6bc48b64dd-vkk6g\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:04.100724 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:04.100034    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-tmp-dir\") pod \"distinct-w3-32c192-kserve-6bc48b64dd-vkk6g\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:04.100724 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:04.100109    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-tls-certs\") pod \"distinct-w3-32c192-kserve-6bc48b64dd-vkk6g\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:04.100724 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:04.100196    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-home\") pod \"distinct-w3-32c192-kserve-6bc48b64dd-vkk6g\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:04.100724 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:04.100284    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-dshm\") pod \"distinct-w3-32c192-kserve-6bc48b64dd-vkk6g\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:04.101384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:04.101361    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-tmp-dir\") pod \"distinct-w3-32c192-kserve-6bc48b64dd-vkk6g\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:04.101645 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:04.101625    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-model-cache\") pod \"distinct-w3-32c192-kserve-6bc48b64dd-vkk6g\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:04.102148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:04.102118    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:55:04.102584 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:04.102188    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm" podUID="c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 29 22:55:04.102875 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:04.102856    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-dshm\") pod \"distinct-w3-32c192-kserve-6bc48b64dd-vkk6g\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:04.102997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:04.102976    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-home\") pod \"distinct-w3-32c192-kserve-6bc48b64dd-vkk6g\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:04.105572 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:04.105553    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-tls-certs\") pod \"distinct-w3-32c192-kserve-6bc48b64dd-vkk6g\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:04.125616 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:04.125592    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znc5d\" (UniqueName: \"kubernetes.io/projected/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-kube-api-access-znc5d\") pod \"distinct-w3-32c192-kserve-6bc48b64dd-vkk6g\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") " pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:04.137959 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:04.137935    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:04.459447 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:04.459418    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"]
Oct 29 22:55:04.462205 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:55:04.462179    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8ada4d0_3756_4ccc_b89c_4b361c4ccc13.slice/crio-fad1a653543d36a80c9de02579dfa7cc189edbe17c23d921e87889376f600cce WatchSource:0}: Error finding container fad1a653543d36a80c9de02579dfa7cc189edbe17c23d921e87889376f600cce: Status 404 returned error can't find the container with id fad1a653543d36a80c9de02579dfa7cc189edbe17c23d921e87889376f600cce
Oct 29 22:55:04.512574 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:04.512540    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" event={"ID":"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13","Type":"ContainerStarted","Data":"fad1a653543d36a80c9de02579dfa7cc189edbe17c23d921e87889376f600cce"}
Oct 29 22:55:05.124466 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:05.124433    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:55:05.125425 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:05.125396    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:55:05.128686 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:05.128665    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 22:55:05.136320 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:05.136291    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:05.520725 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:05.520629    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" event={"ID":"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13","Type":"ContainerStarted","Data":"dc51cb2a3dfcb15cd3f362c831edb227ba0ee579bc12a8ba57f2386b12211073"}
Oct 29 22:55:05.559557 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:05.559506    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podStartSLOduration=2.559490093 podStartE2EDuration="2.559490093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:05.557613736 +0000 UTC m=+708.900562480" watchObservedRunningTime="2026-10-29 22:55:05.559490093 +0000 UTC m=+708.902438837"
Oct 29 22:55:06.138151 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:06.138115    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:06.829089 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:06.829055    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:07.003678 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:07.003643    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:09.105753 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:09.105719    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:10.876667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:10.876636    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:12.466901 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:12.466834    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:14.081531 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:14.081478    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:14.138503 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:14.138473    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:14.152913 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:14.152887    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:14.587592 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:14.587556    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:55:15.664837 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:15.664781    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:17.267004 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:17.266957    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:17.367210 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:55:17.367169    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:55:18.977769 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:18.977733    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:20.669517 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:20.669480    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:22.403138 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:22.403102    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:24.065230 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:24.065188    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:24.185289 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:24.185258    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:25.205971 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:25.205938    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:26.629378 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:26.629333    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:27.489498 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:27.489450    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:29.093703 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:29.093669    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:29.641230 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:29.641197    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:30.765871 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:30.765829    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:32.378853 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:32.378791    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:34.065452 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:34.065396    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:35.761597 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:35.761555    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:37.371788 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:37.371753    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:39.044667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.044635    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:39.186123 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.186081    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"]
Oct 29 22:55:39.190817 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.190782    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.213885 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.213858    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7aeaad\"/\"distinct-w2-7aeaad-kserve-self-signed-certs\""
Oct 29 22:55:39.217907 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.217877    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"]
Oct 29 22:55:39.257083 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.257055    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:39.345215 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.345123    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-tls-certs\") pod \"distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.345215 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.345189    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrfgx\" (UniqueName: \"kubernetes.io/projected/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-kube-api-access-mrfgx\") pod \"distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.345462 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.345357    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-dshm\") pod \"distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.345524 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.345502    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-tmp-dir\") pod \"distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.345581 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.345558    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-home\") pod \"distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.345659 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.345644    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-model-cache\") pod \"distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.426323 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.426288    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"]
Oct 29 22:55:39.431339 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.431316    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.444314 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.444291    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-2077b4-openshift-default-dockercfg-4kftx\""
Oct 29 22:55:39.446924 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.446904    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-home\") pod \"distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.446999 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.446952    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-model-cache\") pod \"distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.446999 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.446992    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-tls-certs\") pod \"distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.447098 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.447034    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrfgx\" (UniqueName: \"kubernetes.io/projected/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-kube-api-access-mrfgx\") pod \"distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.447151 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.447094    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-dshm\") pod \"distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.447209 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.447193    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-tmp-dir\") pod \"distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.447345 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.447321    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-home\") pod \"distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.447405 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.447341    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-model-cache\") pod \"distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.447535 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.447510    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-tmp-dir\") pod \"distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.449232 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.449211    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-dshm\") pod \"distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.449513 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.449495    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-tls-certs\") pod \"distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.453150 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.453129    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"]
Oct 29 22:55:39.468582 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.468563    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrfgx\" (UniqueName: \"kubernetes.io/projected/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-kube-api-access-mrfgx\") pod \"distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.504642 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.504615    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:39.548170 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.548141    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.548288 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.548194    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.548374 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.548345    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.548465 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.548451    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.548501 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.548475    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxhz6\" (UniqueName: \"kubernetes.io/projected/9e6cfa64-3adb-4193-8340-e315e0bbb14e-kube-api-access-lxhz6\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.548501 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.548495    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.548597 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.548551    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.548645 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.548604    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.548681 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.548663    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.649894 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.649862    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.650054 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.649935    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.650054 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.649978    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.650208 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.650074    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.650208 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.650120    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhz6\" (UniqueName: \"kubernetes.io/projected/9e6cfa64-3adb-4193-8340-e315e0bbb14e-kube-api-access-lxhz6\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.650208 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.650153    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.650208 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.650177    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.650427 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.650212    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.650427 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.650241    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.650538 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.650438    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.650538 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.650491    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.650751 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.650728    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.650877 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.650835    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.650937 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.650880    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.652226 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.652205    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.652510 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.652493    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.671504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.671484    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxhz6\" (UniqueName: \"kubernetes.io/projected/9e6cfa64-3adb-4193-8340-e315e0bbb14e-kube-api-access-lxhz6\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.671980 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.671958    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.744733 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.744704    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:39.808647 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:39.807874    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"]
Oct 29 22:55:39.812855 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:55:39.812718    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5ba8e0a_e30f_4663_bb3f_1fab6ec430e9.slice/crio-92d87a1b7449c68ed2ae7a4b3662b3223b9277383ea4f906f93da18bff6fa8b7 WatchSource:0}: Error finding container 92d87a1b7449c68ed2ae7a4b3662b3223b9277383ea4f906f93da18bff6fa8b7: Status 404 returned error can't find the container with id 92d87a1b7449c68ed2ae7a4b3662b3223b9277383ea4f906f93da18bff6fa8b7
Oct 29 22:55:40.065710 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:40.065686    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"]
Oct 29 22:55:40.068227 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:55:40.068195    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6cfa64_3adb_4193_8340_e315e0bbb14e.slice/crio-bd44276229f4c553c42dbbea0392cd71a11c07aee7c12ea5ac9dada32b63f591 WatchSource:0}: Error finding container bd44276229f4c553c42dbbea0392cd71a11c07aee7c12ea5ac9dada32b63f591: Status 404 returned error can't find the container with id bd44276229f4c553c42dbbea0392cd71a11c07aee7c12ea5ac9dada32b63f591
Oct 29 22:55:40.070745 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:40.070713    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:55:40.070846 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:40.070781    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:55:40.070846 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:40.070839    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:55:40.673649 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:40.673615    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:40.750230 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:40.750170    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" event={"ID":"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9","Type":"ContainerStarted","Data":"dfdf37b1df21d6ddbe2f6e0643e1fdbad5e0dd48af35632c362d4a99932aff34"}
Oct 29 22:55:40.750230 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:40.750231    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" event={"ID":"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9","Type":"ContainerStarted","Data":"92d87a1b7449c68ed2ae7a4b3662b3223b9277383ea4f906f93da18bff6fa8b7"}
Oct 29 22:55:40.752360 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:40.752316    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v" event={"ID":"9e6cfa64-3adb-4193-8340-e315e0bbb14e","Type":"ContainerStarted","Data":"d4ca29e35a2af76806e80598b07d1d844a890d002730126f023cc952d5930920"}
Oct 29 22:55:40.752610 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:40.752373    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v" event={"ID":"9e6cfa64-3adb-4193-8340-e315e0bbb14e","Type":"ContainerStarted","Data":"bd44276229f4c553c42dbbea0392cd71a11c07aee7c12ea5ac9dada32b63f591"}
Oct 29 22:55:40.796651 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:40.796590    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podStartSLOduration=1.796571895 podStartE2EDuration="1.796571895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:40.792751003 +0000 UTC m=+744.135699749" watchObservedRunningTime="2026-10-29 22:55:40.796571895 +0000 UTC m=+744.139520651"
Oct 29 22:55:40.834536 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:40.834482    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v" podStartSLOduration=1.8344676519999998 podStartE2EDuration="1.834467652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:40.833580927 +0000 UTC m=+744.176529671" watchObservedRunningTime="2026-10-29 22:55:40.834467652 +0000 UTC m=+744.177416396"
Oct 29 22:55:41.504941 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:41.504904    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:41.749824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:41.747156    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:41.762576 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:41.762478    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:55:41.762576 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:41.762547    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v" podUID="9e6cfa64-3adb-4193-8340-e315e0bbb14e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 29 22:55:42.500860 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:42.500784    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:42.745212 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:42.745178    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:55:42.745594 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:42.745236    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v" podUID="9e6cfa64-3adb-4193-8340-e315e0bbb14e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 29 22:55:43.745363 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:43.745331    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:55:43.745764 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:43.745389    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v" podUID="9e6cfa64-3adb-4193-8340-e315e0bbb14e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 29 22:55:44.251970 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:44.251937    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:44.745647 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:44.745614    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:55:44.746059 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:44.745680    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v" podUID="9e6cfa64-3adb-4193-8340-e315e0bbb14e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 29 22:55:45.745415 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:45.745383    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:55:45.745606 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:45.745450    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v" podUID="9e6cfa64-3adb-4193-8340-e315e0bbb14e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 29 22:55:45.876826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:45.874630    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:46.745467 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:46.745434    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:55:46.745637 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:46.745498    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v" podUID="9e6cfa64-3adb-4193-8340-e315e0bbb14e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 29 22:55:47.367117 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:55:47.367082    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:55:47.577685 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:47.577645    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:47.750367 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:47.750295    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:47.750793 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:47.750774    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:47.751442 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:47.751425    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 22:55:49.125691 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:49.125653    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:49.241603 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:49.241570    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:49.505220 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:49.505127    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:49.518246 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:49.518220    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:49.891348 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:49.891306    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:55:50.859152 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:50.859115    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:51.814192 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:51.814144    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:52.576398 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:52.576359    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:54.407335 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:54.407288    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:56.101931 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:56.101894    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:56.444510 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:56.444475    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:57.949475 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:57.949438    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:59.680557 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:55:59.680520    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:01.261265 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:01.261233    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:01.449408 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:01.449369    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:02.974793 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:02.974756    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:04.448281 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.448243    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"]
Oct 29 22:56:04.452578 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.452553    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.465339 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.465316    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-7aeaad-openshift-default-dockercfg-bxl84\""
Oct 29 22:56:04.482933 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.482902    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"]
Oct 29 22:56:04.570256 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.570224    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:04.613667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.613612    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.613667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.613669    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f56ff730-ff0f-47f0-be77-3bbde2f85301-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.614210 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.613696    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9j2b\" (UniqueName: \"kubernetes.io/projected/f56ff730-ff0f-47f0-be77-3bbde2f85301-kube-api-access-z9j2b\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.614210 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.613975    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.614210 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.614066    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.614210 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.614118    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.614210 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.614177    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.614495 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.614322    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.614495 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.614353    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.716018 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.715926    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.716018 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.715978    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.716264 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.716062    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.716264 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.716101    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.716264 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.716134    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.716264 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.716154    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f56ff730-ff0f-47f0-be77-3bbde2f85301-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.716264 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.716178    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9j2b\" (UniqueName: \"kubernetes.io/projected/f56ff730-ff0f-47f0-be77-3bbde2f85301-kube-api-access-z9j2b\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.716264 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.716250    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.716575 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.716318    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.716629 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.716571    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.716684 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.716661    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.716737 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.716701    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.716784 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.716747    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.717036 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.717017    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f56ff730-ff0f-47f0-be77-3bbde2f85301-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.718968 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.718945    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.719222 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.719202    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.737396 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.737372    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.737937 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.737916    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9j2b\" (UniqueName: \"kubernetes.io/projected/f56ff730-ff0f-47f0-be77-3bbde2f85301-kube-api-access-z9j2b\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:04.766680 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:04.766637    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:05.091498 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:05.091386    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"]
Oct 29 22:56:05.094107 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:56:05.094071    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56ff730_ff0f_47f0_be77_3bbde2f85301.slice/crio-1a240118edb2b76554337af641d90f4dbd7a4ca351fd375e39d57701798988c2 WatchSource:0}: Error finding container 1a240118edb2b76554337af641d90f4dbd7a4ca351fd375e39d57701798988c2: Status 404 returned error can't find the container with id 1a240118edb2b76554337af641d90f4dbd7a4ca351fd375e39d57701798988c2
Oct 29 22:56:05.096545 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:05.096513    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:05.096632 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:05.096578    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:05.096632 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:05.096612    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:05.998763 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:05.998720    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264" event={"ID":"f56ff730-ff0f-47f0-be77-3bbde2f85301","Type":"ContainerStarted","Data":"210dd2597dd76da398bc428e7f16635128420ce3c233f35db49009864f96737f"}
Oct 29 22:56:05.999219 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:05.998787    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264" event={"ID":"f56ff730-ff0f-47f0-be77-3bbde2f85301","Type":"ContainerStarted","Data":"1a240118edb2b76554337af641d90f4dbd7a4ca351fd375e39d57701798988c2"}
Oct 29 22:56:06.091075 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:06.091009    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264" podStartSLOduration=2.090990317 podStartE2EDuration="2.090990317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:06.051599232 +0000 UTC m=+769.394547978" watchObservedRunningTime="2026-10-29 22:56:06.090990317 +0000 UTC m=+769.433939122"
Oct 29 22:56:06.279498 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:06.279389    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:06.784970 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:06.784932    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:56:06.785164 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:06.784994    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264" podUID="f56ff730-ff0f-47f0-be77-3bbde2f85301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:56:06.788450 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:06.788425    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:07.767961 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:07.767915    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:56:07.768502 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:07.767991    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264" podUID="f56ff730-ff0f-47f0-be77-3bbde2f85301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:56:07.988826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:07.986176    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:08.767287 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:08.767247    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:56:08.767571 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:08.767535    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264" podUID="f56ff730-ff0f-47f0-be77-3bbde2f85301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:56:09.773821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:09.773607    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:56:09.773821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:09.773719    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264" podUID="f56ff730-ff0f-47f0-be77-3bbde2f85301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:56:09.889556 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:09.889520    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:10.784066 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:10.783495    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:56:10.784066 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:10.783558    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264" podUID="f56ff730-ff0f-47f0-be77-3bbde2f85301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:56:11.760713 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:11.760671    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:11.767896 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:11.767864    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:56:11.768003 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:11.767931    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264" podUID="f56ff730-ff0f-47f0-be77-3bbde2f85301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:56:12.777071 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:12.777040    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:12.777556 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:12.777535    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:12.778242 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:12.778220    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 22:56:13.381772 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:13.381696    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:13.963923 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:13.963884    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:15.081345 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:15.081311    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:15.340061 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:15.339986    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:16.774516 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:16.774485    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:17.367663 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:56:17.367629    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:56:18.343317 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:18.343268    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:19.881538 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:19.881503    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:21.523204 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:21.523168    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:23.051926 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:23.051885    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:24.226650 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:24.226617    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:24.689526 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:24.689491    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:26.463297 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:26.463257    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:27.747886 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:27.747847    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"]
Oct 29 22:56:27.754130 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:27.754103    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:27.773173 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:27.773148    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"]
Oct 29 22:56:27.779764 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:27.779731    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7aeaad\"/\"distinct-2-w2-7aeaad-kserve-self-signed-certs\""
Oct 29 22:56:27.814107 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:27.814080    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:27.918143 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:27.918104    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-home\") pod \"distinct-2-w2-7aeaad-kserve-647985bff-67bft\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:27.918143 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:27.918144    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-dshm\") pod \"distinct-2-w2-7aeaad-kserve-647985bff-67bft\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:27.918404 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:27.918294    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-tmp-dir\") pod \"distinct-2-w2-7aeaad-kserve-647985bff-67bft\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:27.918404 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:27.918319    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-tls-certs\") pod \"distinct-2-w2-7aeaad-kserve-647985bff-67bft\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:27.918503 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:27.918415    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqbjc\" (UniqueName: \"kubernetes.io/projected/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-kube-api-access-pqbjc\") pod \"distinct-2-w2-7aeaad-kserve-647985bff-67bft\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:27.918583 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:27.918562    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-model-cache\") pod \"distinct-2-w2-7aeaad-kserve-647985bff-67bft\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:28.020248 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:28.020162    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-model-cache\") pod \"distinct-2-w2-7aeaad-kserve-647985bff-67bft\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:28.020406 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:28.020308    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-home\") pod \"distinct-2-w2-7aeaad-kserve-647985bff-67bft\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:28.020406 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:28.020329    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-dshm\") pod \"distinct-2-w2-7aeaad-kserve-647985bff-67bft\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:28.020406 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:28.020351    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-tmp-dir\") pod \"distinct-2-w2-7aeaad-kserve-647985bff-67bft\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:28.020406 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:28.020367    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-tls-certs\") pod \"distinct-2-w2-7aeaad-kserve-647985bff-67bft\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:28.020406 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:28.020403    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqbjc\" (UniqueName: \"kubernetes.io/projected/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-kube-api-access-pqbjc\") pod \"distinct-2-w2-7aeaad-kserve-647985bff-67bft\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:28.020678 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:28.020640    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-model-cache\") pod \"distinct-2-w2-7aeaad-kserve-647985bff-67bft\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:28.020733 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:28.020682    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-home\") pod \"distinct-2-w2-7aeaad-kserve-647985bff-67bft\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:28.020733 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:28.020719    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-tmp-dir\") pod \"distinct-2-w2-7aeaad-kserve-647985bff-67bft\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:28.022605 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:28.022582    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-dshm\") pod \"distinct-2-w2-7aeaad-kserve-647985bff-67bft\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:28.022873 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:28.022856    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-tls-certs\") pod \"distinct-2-w2-7aeaad-kserve-647985bff-67bft\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:28.042137 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:28.042111    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqbjc\" (UniqueName: \"kubernetes.io/projected/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-kube-api-access-pqbjc\") pod \"distinct-2-w2-7aeaad-kserve-647985bff-67bft\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") " pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:28.070485 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:28.070450    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:28.382521 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:28.382495    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"]
Oct 29 22:56:28.385424 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:56:28.385393    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22d79ec5_027d_4dc6_b4ac_ec7e018c039a.slice/crio-0501c3f90a31f6bbd1e7a3afd1805bde45a3b46291ce0f75a256daa44ab98201 WatchSource:0}: Error finding container 0501c3f90a31f6bbd1e7a3afd1805bde45a3b46291ce0f75a256daa44ab98201: Status 404 returned error can't find the container with id 0501c3f90a31f6bbd1e7a3afd1805bde45a3b46291ce0f75a256daa44ab98201
Oct 29 22:56:28.412542 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:28.412501    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:29.263959 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:29.263911    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" event={"ID":"22d79ec5-027d-4dc6-b4ac-ec7e018c039a","Type":"ContainerStarted","Data":"d4e9a5b5b2c7445d269d14c0c32703b28582924df4662ba2be894ddee81a08da"}
Oct 29 22:56:29.264478 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:29.263970    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" event={"ID":"22d79ec5-027d-4dc6-b4ac-ec7e018c039a","Type":"ContainerStarted","Data":"0501c3f90a31f6bbd1e7a3afd1805bde45a3b46291ce0f75a256daa44ab98201"}
Oct 29 22:56:29.307395 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:29.307325    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podStartSLOduration=2.307306118 podStartE2EDuration="2.307306118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:29.305025044 +0000 UTC m=+792.647973817" watchObservedRunningTime="2026-10-29 22:56:29.307306118 +0000 UTC m=+792.650254864"
Oct 29 22:56:30.071397 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:30.071344    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:30.841049 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:30.841010    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:33.181271 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:33.181231    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:35.167071 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:35.167034    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:36.883068 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:36.883031    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:38.070868 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:38.070831    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:38.084873 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:38.084847    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:38.385265 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:38.385239    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:56:38.697783 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:38.697707    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:40.388876 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:40.388797    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:41.395882 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.395851    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"]
Oct 29 22:56:41.400893 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.400869    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.420492 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.420463    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"]
Oct 29 22:56:41.426620 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.426591    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-99dfb0\"/\"openshift-service-ca.crt\""
Oct 29 22:56:41.426620 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.426601    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-99dfb0\"/\"kube-root-ca.crt\""
Oct 29 22:56:41.426810 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.426747    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-99dfb0\"/\"default-dockercfg-nf4k7\""
Oct 29 22:56:41.426860 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.426795    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-99dfb0\"/\"distinct-w1-99dfb0-kserve-self-signed-certs\""
Oct 29 22:56:41.449786 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.449746    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrdxm\" (UniqueName: \"kubernetes.io/projected/3c62dd36-e795-4778-902e-a990d1998123-kube-api-access-rrdxm\") pod \"distinct-w1-99dfb0-kserve-59b7978874-ngqmk\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.450033 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.449861    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-tmp-dir\") pod \"distinct-w1-99dfb0-kserve-59b7978874-ngqmk\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.450033 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.449909    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-dshm\") pod \"distinct-w1-99dfb0-kserve-59b7978874-ngqmk\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.450033 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.449944    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-home\") pod \"distinct-w1-99dfb0-kserve-59b7978874-ngqmk\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.450033 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.449988    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c62dd36-e795-4778-902e-a990d1998123-tls-certs\") pod \"distinct-w1-99dfb0-kserve-59b7978874-ngqmk\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.450269 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.450109    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-model-cache\") pod \"distinct-w1-99dfb0-kserve-59b7978874-ngqmk\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.472025 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.471989    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:41.551586 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.551556    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrdxm\" (UniqueName: \"kubernetes.io/projected/3c62dd36-e795-4778-902e-a990d1998123-kube-api-access-rrdxm\") pod \"distinct-w1-99dfb0-kserve-59b7978874-ngqmk\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.551753 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.551627    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-tmp-dir\") pod \"distinct-w1-99dfb0-kserve-59b7978874-ngqmk\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.551753 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.551671    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-dshm\") pod \"distinct-w1-99dfb0-kserve-59b7978874-ngqmk\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.551753 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.551702    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-home\") pod \"distinct-w1-99dfb0-kserve-59b7978874-ngqmk\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.551753 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.551746    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c62dd36-e795-4778-902e-a990d1998123-tls-certs\") pod \"distinct-w1-99dfb0-kserve-59b7978874-ngqmk\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.551987 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.551862    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-model-cache\") pod \"distinct-w1-99dfb0-kserve-59b7978874-ngqmk\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.552130 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.552106    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-tmp-dir\") pod \"distinct-w1-99dfb0-kserve-59b7978874-ngqmk\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.552194 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.552119    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-home\") pod \"distinct-w1-99dfb0-kserve-59b7978874-ngqmk\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.552317 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.552290    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-model-cache\") pod \"distinct-w1-99dfb0-kserve-59b7978874-ngqmk\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.553978 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.553952    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-dshm\") pod \"distinct-w1-99dfb0-kserve-59b7978874-ngqmk\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.554230 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.554214    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c62dd36-e795-4778-902e-a990d1998123-tls-certs\") pod \"distinct-w1-99dfb0-kserve-59b7978874-ngqmk\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.573298 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.573271    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrdxm\" (UniqueName: \"kubernetes.io/projected/3c62dd36-e795-4778-902e-a990d1998123-kube-api-access-rrdxm\") pod \"distinct-w1-99dfb0-kserve-59b7978874-ngqmk\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") " pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:41.716685 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:41.716599    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:42.024378 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:42.024348    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"]
Oct 29 22:56:42.026350 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:56:42.026322    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c62dd36_e795_4778_902e_a990d1998123.slice/crio-d08dc7e729173fd949915c2968d1246aec510eef193b206348112a8b95de8a98 WatchSource:0}: Error finding container d08dc7e729173fd949915c2968d1246aec510eef193b206348112a8b95de8a98: Status 404 returned error can't find the container with id d08dc7e729173fd949915c2968d1246aec510eef193b206348112a8b95de8a98
Oct 29 22:56:42.176709 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:42.176674    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:42.412030 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:42.411998    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" event={"ID":"3c62dd36-e795-4778-902e-a990d1998123","Type":"ContainerStarted","Data":"34af99d2461bcebede0b4d541f2a4ec07b87cfe75583e99f9e73ac827c1ff7fa"}
Oct 29 22:56:42.412430 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:42.412037    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" event={"ID":"3c62dd36-e795-4778-902e-a990d1998123","Type":"ContainerStarted","Data":"d08dc7e729173fd949915c2968d1246aec510eef193b206348112a8b95de8a98"}
Oct 29 22:56:42.460175 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:42.460119    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podStartSLOduration=1.460103438 podStartE2EDuration="1.460103438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:42.449735443 +0000 UTC m=+805.792684188" watchObservedRunningTime="2026-10-29 22:56:42.460103438 +0000 UTC m=+805.803052183"
Oct 29 22:56:42.717207 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:42.717122    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:43.879576 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:43.879526    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:44.583087 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:44.583044    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:45.752175 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:45.752137    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:45.803823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:45.803481    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"]
Oct 29 22:56:45.812841 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:45.810047    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:45.837020 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:45.836986    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"]
Oct 29 22:56:45.917129 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:45.917077    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:45.917322 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:45.917152    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:45.917322 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:45.917255    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:45.917454 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:45.917357    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:45.917454 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:45.917410    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:45.917454 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:45.917449    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wklkj\" (UniqueName: \"kubernetes.io/projected/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-kube-api-access-wklkj\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:45.917627 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:45.917505    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:45.917697 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:45.917673    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:45.917757 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:45.917724    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.019697 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.019583    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.019697 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.019663    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wklkj\" (UniqueName: \"kubernetes.io/projected/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-kube-api-access-wklkj\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.019697 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.019691    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.020028 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.019834    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.020028 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.019874    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.020028 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.019953    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.020028 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.020000    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.020259 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.020037    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.020259 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.020101    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.020259 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.020146    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.020259 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.020195    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.020473 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.020451    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.020899 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.020869    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.021049 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.020992    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.023060 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.023026    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.023340 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.023313    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.041363 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.041332    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.041488 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.041473    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wklkj\" (UniqueName: \"kubernetes.io/projected/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-kube-api-access-wklkj\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.137425 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.137387    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:46.505466 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.505440    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"]
Oct 29 22:56:46.507786 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:56:46.507756    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb3a728_4b70_4a8b_bdd3_6fe99b4fa3d4.slice/crio-1456a15fef9c59d3564eca75aefe1f8eee4c75ced9d7cf1b603109cd4f5fa58e WatchSource:0}: Error finding container 1456a15fef9c59d3564eca75aefe1f8eee4c75ced9d7cf1b603109cd4f5fa58e: Status 404 returned error can't find the container with id 1456a15fef9c59d3564eca75aefe1f8eee4c75ced9d7cf1b603109cd4f5fa58e
Oct 29 22:56:46.510472 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.510438    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:46.510575 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.510517    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:46.510575 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:46.510548    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:47.396739 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:56:47.395219    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:56:47.402874 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:47.402844    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:47.458235 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:47.455837    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh" event={"ID":"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4","Type":"ContainerStarted","Data":"9d19fcdd5fa352e69e566e0d23f5e2e66c30c5ef90ed5d7baa0604b99c47d43c"}
Oct 29 22:56:47.458235 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:47.455919    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh" event={"ID":"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4","Type":"ContainerStarted","Data":"1456a15fef9c59d3564eca75aefe1f8eee4c75ced9d7cf1b603109cd4f5fa58e"}
Oct 29 22:56:47.530066 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:47.529989    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh" podStartSLOduration=2.529970633 podStartE2EDuration="2.529970633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:47.529187555 +0000 UTC m=+810.872136302" watchObservedRunningTime="2026-10-29 22:56:47.529970633 +0000 UTC m=+810.872919379"
Oct 29 22:56:48.138490 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:48.138458    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:48.140020 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:48.139993    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 29 22:56:48.140151 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:48.140047    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh" podUID="fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 29 22:56:48.852921 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:48.852882    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"]
Oct 29 22:56:48.853332 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:48.853107    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc" podUID="42695ed6-95ae-457a-8822-8c9e303d06b4" containerName="istio-proxy" containerID="cri-o://019f4c1065a6f1f532e07f6dc23d6f2ecaac57d95542f16fbf06f6e052f1d4c6" gracePeriod=30
Oct 29 22:56:48.987552 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:48.987508    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:49.138243 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:49.138212    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 29 22:56:49.138409 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:49.138274    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh" podUID="fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 29 22:56:50.138408 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:50.138377    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 29 22:56:50.138829 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:50.138436    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh" podUID="fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 29 22:56:51.138370 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.138333    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 29 22:56:51.138537 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.138390    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh" podUID="fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 29 22:56:51.717726 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.717692    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:51.728340 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.728310    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"]
Oct 29 22:56:51.732743 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.732721    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:51.732895 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.732796    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.805336 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.802931    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"]
Oct 29 22:56:51.828240 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.828197    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.829928 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.828261    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.829928 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.828308    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.829928 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.828350    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.829928 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.828389    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-264f7\" (UniqueName: \"kubernetes.io/projected/204e9634-b4e0-477d-8114-10fd7ecf0f72-kube-api-access-264f7\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.829928 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.828441    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.829928 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.828514    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.829928 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.828549    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204e9634-b4e0-477d-8114-10fd7ecf0f72-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.829928 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.828996    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.930430 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.930390    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.930585 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.930455    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.930585 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.930480    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204e9634-b4e0-477d-8114-10fd7ecf0f72-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.930585 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.930560    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.930863 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.930840    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.930986 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.930960    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.931167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.931147    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.931234 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.931183    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.931234 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.931213    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.931330 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.931242    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.931330 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.931269    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-264f7\" (UniqueName: \"kubernetes.io/projected/204e9634-b4e0-477d-8114-10fd7ecf0f72-kube-api-access-264f7\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.931330 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.931299    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204e9634-b4e0-477d-8114-10fd7ecf0f72-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.931612 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.931575    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.931707 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.931629    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.933113 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.933080    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.933251 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.933233    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.952713 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.952691    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-264f7\" (UniqueName: \"kubernetes.io/projected/204e9634-b4e0-477d-8114-10fd7ecf0f72-kube-api-access-264f7\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:51.952795 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:51.952697    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:52.110263 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:52.110181    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:52.118067 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:52.118039    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:52.138439 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:52.138415    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 29 22:56:52.138545 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:52.138474    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh" podUID="fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 29 22:56:52.295668 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:52.295585    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:52.469332 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:52.469308    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"]
Oct 29 22:56:52.471466 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:56:52.471438    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204e9634_b4e0_477d_8114_10fd7ecf0f72.slice/crio-ac37a35e2d3e2b3b4e16dcec839d4caf836a3fd5306ad8192ea980b3f13c01a8 WatchSource:0}: Error finding container ac37a35e2d3e2b3b4e16dcec839d4caf836a3fd5306ad8192ea980b3f13c01a8: Status 404 returned error can't find the container with id ac37a35e2d3e2b3b4e16dcec839d4caf836a3fd5306ad8192ea980b3f13c01a8
Oct 29 22:56:52.474142 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:52.474108    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:52.474245 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:52.474175    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:52.474245 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:52.474203    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:52.521527 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:52.521501    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2" event={"ID":"204e9634-b4e0-477d-8114-10fd7ecf0f72","Type":"ContainerStarted","Data":"ac37a35e2d3e2b3b4e16dcec839d4caf836a3fd5306ad8192ea980b3f13c01a8"}
Oct 29 22:56:52.533202 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:52.533162    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:56:53.137931 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:53.137895    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 29 22:56:53.138103 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:53.137963    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh" podUID="fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 29 22:56:53.532357 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:53.532252    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2" event={"ID":"204e9634-b4e0-477d-8114-10fd7ecf0f72","Type":"ContainerStarted","Data":"5b1e1e76ebe8ca74d49290ba27328a8c300cfcadcde8e9612f550e05cfb9abc9"}
Oct 29 22:56:53.600377 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:53.600319    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2" podStartSLOduration=2.600303508 podStartE2EDuration="2.600303508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:53.585263903 +0000 UTC m=+816.928212648" watchObservedRunningTime="2026-10-29 22:56:53.600303508 +0000 UTC m=+816.943252252"
Oct 29 22:56:54.116164 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.116110    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:56:54.127415 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.126458    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:56:54.127415 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.126518    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2" podUID="204e9634-b4e0-477d-8114-10fd7ecf0f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:56:54.153002 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.151551    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:54.153002 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.152689    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:54.154831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.154778    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 22:56:54.196286 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.196257    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:54.218641 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.218616    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:56:54.291518 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.291483    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-envoy\") pod \"42695ed6-95ae-457a-8822-8c9e303d06b4\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") "
Oct 29 22:56:54.291686 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.291570    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-credential-socket\") pod \"42695ed6-95ae-457a-8822-8c9e303d06b4\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") "
Oct 29 22:56:54.291686 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.291629    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42695ed6-95ae-457a-8822-8c9e303d06b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42695ed6-95ae-457a-8822-8c9e303d06b4-istiod-ca-cert\") pod \"42695ed6-95ae-457a-8822-8c9e303d06b4\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") "
Oct 29 22:56:54.291686 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.291681    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-workload-certs\") pod \"42695ed6-95ae-457a-8822-8c9e303d06b4\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") "
Oct 29 22:56:54.291831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.291716    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-token\" (UniqueName: \"kubernetes.io/projected/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-token\") pod \"42695ed6-95ae-457a-8822-8c9e303d06b4\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") "
Oct 29 22:56:54.291831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.291750    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42695ed6-95ae-457a-8822-8c9e303d06b4-kube-api-access-2b4rz\" (UniqueName: \"kubernetes.io/projected/42695ed6-95ae-457a-8822-8c9e303d06b4-kube-api-access-2b4rz\") pod \"42695ed6-95ae-457a-8822-8c9e303d06b4\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") "
Oct 29 22:56:54.291831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.291788    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-podinfo\") pod \"42695ed6-95ae-457a-8822-8c9e303d06b4\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") "
Oct 29 22:56:54.291995 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.291852    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-data\") pod \"42695ed6-95ae-457a-8822-8c9e303d06b4\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") "
Oct 29 22:56:54.291995 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.291881    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-credential-socket" pod "42695ed6-95ae-457a-8822-8c9e303d06b4" (UID: "42695ed6-95ae-457a-8822-8c9e303d06b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:56:54.291995 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.291902    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-workload-socket\") pod \"42695ed6-95ae-457a-8822-8c9e303d06b4\" (UID: \"42695ed6-95ae-457a-8822-8c9e303d06b4\") "
Oct 29 22:56:54.291995 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.291977    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-workload-certs" pod "42695ed6-95ae-457a-8822-8c9e303d06b4" (UID: "42695ed6-95ae-457a-8822-8c9e303d06b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:56:54.292209 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.292149    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42695ed6-95ae-457a-8822-8c9e303d06b4-istiod-ca-cert" pod "42695ed6-95ae-457a-8822-8c9e303d06b4" (UID: "42695ed6-95ae-457a-8822-8c9e303d06b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:56:54.292290 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.292262    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-workload-socket" pod "42695ed6-95ae-457a-8822-8c9e303d06b4" (UID: "42695ed6-95ae-457a-8822-8c9e303d06b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:56:54.292352 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.292338    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-data" pod "42695ed6-95ae-457a-8822-8c9e303d06b4" (UID: "42695ed6-95ae-457a-8822-8c9e303d06b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:56:54.292547 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.292521    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:56:54.292631 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.292549    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:56:54.292631 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.292564    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:56:54.292631 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.292578    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42695ed6-95ae-457a-8822-8c9e303d06b4-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:56:54.292631 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.292591    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:56:54.293853 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.293790    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-envoy" pod "42695ed6-95ae-457a-8822-8c9e303d06b4" (UID: "42695ed6-95ae-457a-8822-8c9e303d06b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:56:54.294185 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.294163    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42695ed6-95ae-457a-8822-8c9e303d06b4-kube-api-access-2b4rz" pod "42695ed6-95ae-457a-8822-8c9e303d06b4" (UID: "42695ed6-95ae-457a-8822-8c9e303d06b4"). InnerVolumeSpecName "kube-api-access-2b4rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:54.294275 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.294197    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-token" pod "42695ed6-95ae-457a-8822-8c9e303d06b4" (UID: "42695ed6-95ae-457a-8822-8c9e303d06b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:54.294275 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.294206    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-podinfo" pod "42695ed6-95ae-457a-8822-8c9e303d06b4" (UID: "42695ed6-95ae-457a-8822-8c9e303d06b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:56:54.394013 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.393921    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:56:54.394013 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.393959    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:56:54.394013 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.393977    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b4rz\" (UniqueName: \"kubernetes.io/projected/42695ed6-95ae-457a-8822-8c9e303d06b4-kube-api-access-2b4rz\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:56:54.394013 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.393992    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42695ed6-95ae-457a-8822-8c9e303d06b4-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 22:56:54.540866 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.540828    5428 generic.go:356] "Generic (PLEG): container finished" podID="42695ed6-95ae-457a-8822-8c9e303d06b4" containerID="019f4c1065a6f1f532e07f6dc23d6f2ecaac57d95542f16fbf06f6e052f1d4c6" exitCode=0
Oct 29 22:56:54.541255 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.540936    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc" event={"ID":"42695ed6-95ae-457a-8822-8c9e303d06b4","Type":"ContainerDied","Data":"019f4c1065a6f1f532e07f6dc23d6f2ecaac57d95542f16fbf06f6e052f1d4c6"}
Oct 29 22:56:54.541255 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.540985    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc" event={"ID":"42695ed6-95ae-457a-8822-8c9e303d06b4","Type":"ContainerDied","Data":"93193bd0f0966b1c799bf1b87bef44938aa30bb6eccce25565ca7da59c7c001a"}
Oct 29 22:56:54.541255 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.540989    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"
Oct 29 22:56:54.541255 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.541007    5428 scope.go:122] "RemoveContainer" containerID="019f4c1065a6f1f532e07f6dc23d6f2ecaac57d95542f16fbf06f6e052f1d4c6"
Oct 29 22:56:54.551740 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.551717    5428 scope.go:122] "RemoveContainer" containerID="019f4c1065a6f1f532e07f6dc23d6f2ecaac57d95542f16fbf06f6e052f1d4c6"
Oct 29 22:56:54.552869 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:56:54.552622    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"019f4c1065a6f1f532e07f6dc23d6f2ecaac57d95542f16fbf06f6e052f1d4c6\": container with ID starting with 019f4c1065a6f1f532e07f6dc23d6f2ecaac57d95542f16fbf06f6e052f1d4c6 not found: ID does not exist" containerID="019f4c1065a6f1f532e07f6dc23d6f2ecaac57d95542f16fbf06f6e052f1d4c6"
Oct 29 22:56:54.552869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.552647    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"019f4c1065a6f1f532e07f6dc23d6f2ecaac57d95542f16fbf06f6e052f1d4c6"} err="failed to get container status \"019f4c1065a6f1f532e07f6dc23d6f2ecaac57d95542f16fbf06f6e052f1d4c6\": rpc error: code = NotFound desc = could not find container \"019f4c1065a6f1f532e07f6dc23d6f2ecaac57d95542f16fbf06f6e052f1d4c6\": container with ID starting with 019f4c1065a6f1f532e07f6dc23d6f2ecaac57d95542f16fbf06f6e052f1d4c6 not found: ID does not exist"
Oct 29 22:56:54.598820 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.598775    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"]
Oct 29 22:56:54.602383 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.602352    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-fssgc"]
Oct 29 22:56:54.655754 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.655677    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"]
Oct 29 22:56:54.656335 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.656315    5428 cpu_manager.go:414] "containerMap: removing container" podUID="42695ed6-95ae-457a-8822-8c9e303d06b4" containerName="istio-proxy"
Oct 29 22:56:54.656436 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.656338    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42695ed6-95ae-457a-8822-8c9e303d06b4" containerName="istio-proxy"
Oct 29 22:56:54.656965 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.656490    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="42695ed6-95ae-457a-8822-8c9e303d06b4" containerName="istio-proxy"
Oct 29 22:56:54.692638 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.692609    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.697258 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.697234    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"]
Oct 29 22:56:54.791651 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.791616    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42695ed6-95ae-457a-8822-8c9e303d06b4" path="/var/lib/kubelet/pods/42695ed6-95ae-457a-8822-8c9e303d06b4/volumes"
Oct 29 22:56:54.800874 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.800846    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fwdv\" (UniqueName: \"kubernetes.io/projected/5b094a72-da28-4b91-9dec-f6d9d55543b9-kube-api-access-9fwdv\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.801020 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.800916    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.801086 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.801020    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.801086 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.801064    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.801197 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.801180    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.801257 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.801231    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.801348 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.801324    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b094a72-da28-4b91-9dec-f6d9d55543b9-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.801386 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.801360    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.801437 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.801407    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.902430 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.902391    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.902595 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.902448    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.902595 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.902481    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b094a72-da28-4b91-9dec-f6d9d55543b9-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.902595 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.902500    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.902595 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.902525    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.902840 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.902626    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fwdv\" (UniqueName: \"kubernetes.io/projected/5b094a72-da28-4b91-9dec-f6d9d55543b9-kube-api-access-9fwdv\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.902840 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.902692    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.902840 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.902753    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.902840 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.902781    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.903040 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.902876    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.903098 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.903080    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.903200 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.903174    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.903688 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.903246    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.903688 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.903259    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b094a72-da28-4b91-9dec-f6d9d55543b9-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.905126 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.905103    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.905422 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.905401    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.925885 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.925836    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:54.927491 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:54.927467    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fwdv\" (UniqueName: \"kubernetes.io/projected/5b094a72-da28-4b91-9dec-f6d9d55543b9-kube-api-access-9fwdv\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:55.009953 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:55.009923    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:55.111558 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:55.111233    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:56:55.111558 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:55.111280    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2" podUID="204e9634-b4e0-477d-8114-10fd7ecf0f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:56:55.388153 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:55.388128    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"]
Oct 29 22:56:55.389963 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:56:55.389932    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b094a72_da28_4b91_9dec_f6d9d55543b9.slice/crio-c6f03efee1c167a63045c4bd3ef6a48e2ca7cfc940c39aeaf8dcaa115a3423c3 WatchSource:0}: Error finding container c6f03efee1c167a63045c4bd3ef6a48e2ca7cfc940c39aeaf8dcaa115a3423c3: Status 404 returned error can't find the container with id c6f03efee1c167a63045c4bd3ef6a48e2ca7cfc940c39aeaf8dcaa115a3423c3
Oct 29 22:56:55.392600 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:55.392567    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:55.392691 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:55.392634    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:55.392691 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:55.392665    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:55.553537 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:55.553495    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f" event={"ID":"5b094a72-da28-4b91-9dec-f6d9d55543b9","Type":"ContainerStarted","Data":"661e425d0576452c701691e0015301a911143ab9a231b2b454f1609588c7df87"}
Oct 29 22:56:55.554008 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:55.553561    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f" event={"ID":"5b094a72-da28-4b91-9dec-f6d9d55543b9","Type":"ContainerStarted","Data":"c6f03efee1c167a63045c4bd3ef6a48e2ca7cfc940c39aeaf8dcaa115a3423c3"}
Oct 29 22:56:55.614032 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:55.613974    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f" podStartSLOduration=1.61395449 podStartE2EDuration="1.61395449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:55.60985269 +0000 UTC m=+818.952801430" watchObservedRunningTime="2026-10-29 22:56:55.61395449 +0000 UTC m=+818.956903238"
Oct 29 22:56:55.890388 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:55.890354    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:56.010990 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:56.010955    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:56:56.012780 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:56.012753    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:56:56.012914 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:56.012839    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f" podUID="5b094a72-da28-4b91-9dec-f6d9d55543b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:56:56.111453 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:56.111418    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:56:56.111619 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:56.111474    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2" podUID="204e9634-b4e0-477d-8114-10fd7ecf0f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:56:57.010501 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:57.010464    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:56:57.011018 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:57.010578    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f" podUID="5b094a72-da28-4b91-9dec-f6d9d55543b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:56:57.113820 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:57.112931    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:56:57.113820 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:57.112994    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2" podUID="204e9634-b4e0-477d-8114-10fd7ecf0f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:56:57.931992 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:57.931882    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:58.021753 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.021718    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:56:58.022341 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.022167    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f" podUID="5b094a72-da28-4b91-9dec-f6d9d55543b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:56:58.113219 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.113182    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:56:58.113483 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.113457    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2" podUID="204e9634-b4e0-477d-8114-10fd7ecf0f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:56:58.209143 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.209058    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"]
Oct 29 22:56:58.233157 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.233130    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"]
Oct 29 22:56:58.233471 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.233415    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.249420 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.249394    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.249549 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.249424    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.249549 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.249458    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.249658 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.249618    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.249713 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.249664    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6d84\" (UniqueName: \"kubernetes.io/projected/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-kube-api-access-b6d84\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.249773 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.249729    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.249845 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.249816    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.249911 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.249855    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.249963 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.249910    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.350983 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.350943    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.350983 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.350986    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.351184 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.351021    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.351184 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.351066    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.351184 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.351084    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6d84\" (UniqueName: \"kubernetes.io/projected/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-kube-api-access-b6d84\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.351184 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.351121    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.351184 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.351161    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.351184 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.351183    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.351382 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.351212    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.351382 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.351370    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.351572 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.351549    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.351838 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.351781    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.352121 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.352099    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.352405 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.352388    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.353372 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.353346    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.353516 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.353497    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.395694 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.395654    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6d84\" (UniqueName: \"kubernetes.io/projected/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-kube-api-access-b6d84\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.396415 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.396339    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.549130 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.549042    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:56:58.904464 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.904440    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"]
Oct 29 22:56:58.907355 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:56:58.907329    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e3c3eb_c2d6_4c8d_b1db_c953a8964b28.slice/crio-600f7461c71415b3afcd877ad02cd72ca3ed3bba90c585954edeffe489d276e3 WatchSource:0}: Error finding container 600f7461c71415b3afcd877ad02cd72ca3ed3bba90c585954edeffe489d276e3: Status 404 returned error can't find the container with id 600f7461c71415b3afcd877ad02cd72ca3ed3bba90c585954edeffe489d276e3
Oct 29 22:56:58.909982 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.909952    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:58.910040 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.910018    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:58.910096 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:58.910046    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:56:59.010897 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:59.010870    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:56:59.011001 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:59.010926    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f" podUID="5b094a72-da28-4b91-9dec-f6d9d55543b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:56:59.111291 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:59.110862    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:56:59.111291 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:59.110921    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2" podUID="204e9634-b4e0-477d-8114-10fd7ecf0f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:56:59.614658 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:59.614623    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr" event={"ID":"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28","Type":"ContainerStarted","Data":"cc0384e2c90a2ac302230f7847b2f40f6d2dfb74e7d38c2f56219ade6454ce63"}
Oct 29 22:56:59.614869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:59.614674    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr" event={"ID":"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28","Type":"ContainerStarted","Data":"600f7461c71415b3afcd877ad02cd72ca3ed3bba90c585954edeffe489d276e3"}
Oct 29 22:56:59.674493 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:59.674427    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr" podStartSLOduration=1.674408603 podStartE2EDuration="1.674408603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:59.65807643 +0000 UTC m=+823.001025176" watchObservedRunningTime="2026-10-29 22:56:59.674408603 +0000 UTC m=+823.017357349"
Oct 29 22:56:59.695830 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:56:59.691271    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:00.010907 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:00.010441    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:57:00.010907 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:00.010523    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f" podUID="5b094a72-da28-4b91-9dec-f6d9d55543b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:57:00.243125 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:00.243090    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:57:00.243910 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:00.243884    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:57:00.244692 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:00.244673    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 22:57:00.550077 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:00.549269    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:57:00.550830 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:00.550785    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 29 22:57:00.550886 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:00.550856    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr" podUID="53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 29 22:57:01.011821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:01.011334    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:57:01.011821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:01.011391    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f" podUID="5b094a72-da28-4b91-9dec-f6d9d55543b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:57:01.549897 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:01.549864    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 29 22:57:01.550301 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:01.549916    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr" podUID="53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 29 22:57:01.600825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:01.600298    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:02.010414 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:02.010362    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:57:02.010621 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:02.010449    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f" podUID="5b094a72-da28-4b91-9dec-f6d9d55543b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:57:02.555834 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:02.553764    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 29 22:57:02.555834 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:02.553911    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr" podUID="53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 29 22:57:02.896859 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:02.891663    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:03.020823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:03.019935    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:57:03.020823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:03.020007    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f" podUID="5b094a72-da28-4b91-9dec-f6d9d55543b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:57:03.556978 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:03.554491    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 29 22:57:03.556978 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:03.554581    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr" podUID="53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 29 22:57:04.222896 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:04.219924    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:04.378824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:04.377901    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:04.531880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:04.530897    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:04.550825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:04.550435    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 29 22:57:04.550825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:04.550489    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr" podUID="53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 29 22:57:04.578822 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:04.577889    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:04.891481 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:04.891424    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:57:04.978841 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:04.952030    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:05.105057 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.105014    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:57:05.112824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.105292    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f" podUID="5b094a72-da28-4b91-9dec-f6d9d55543b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:57:05.123825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.118091    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 29 22:57:05.149872 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.149285    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:05.221155 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.221112    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:57:05.244822 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.223547    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:57:05.441882 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.441703    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:05.466826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.442505    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:57:05.485815 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.485760    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:05.697886 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.696121    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 29 22:57:05.725873 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.712288    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 29 22:57:05.725873 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.712356    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr" podUID="53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 29 22:57:05.725873 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.712453    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:57:05.725873 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.714672    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:57:05.725873 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.724659    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:57:05.960899 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.950908    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:57:05.960899 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.951704    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:57:05.960899 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.954177    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:05.960899 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:05.955505    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:57:06.402899 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:06.384644    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:06.402899 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:06.397401    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded"
Oct 29 22:57:06.406736 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:06.406193    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:57:06.480492 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:06.480454    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:06.480754 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:06.480724    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm" podUID="c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:06.480971 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:06.480948    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:57:06.486877 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:06.486854    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:57:06.506848 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:06.505267    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:06.565833 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:06.564981    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded"
Oct 29 22:57:06.678581 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:06.678479    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 29 22:57:06.678878 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:06.678845    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr" podUID="53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 29 22:57:06.679201 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:06.679179    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:06.682822 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:06.679342    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd" podUID="ae1fd856-9cb0-4d65-bc25-49aefe12e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:06.703390 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:06.703355    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:57:06.719194 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:06.719154    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:57:06.965052 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:06.959240    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.014254 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:06.968285    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:07.014536 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:07.014501    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f" podUID="5b094a72-da28-4b91-9dec-f6d9d55543b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.068244 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:06.998052    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.423369 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:07.423340    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:57:07.426851 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:07.423718    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:57:07.453929 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:07.453900    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:57:07.463147 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:07.463121    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:57:07.476711 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:07.476490    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:57:07.550287 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:07.550255    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 29 22:57:07.550525 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:07.550501    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr" podUID="53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 29 22:57:07.686093 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:07.685996    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:57:07.687754 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:07.687733    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:57:07.694162 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:07.694137    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 22:57:08.357655 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:08.357616    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:08.489549 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:08.489516    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:08.549751 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:08.549705    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 29 22:57:08.549918 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:08.549775    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr" podUID="53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 29 22:57:08.632284 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:08.632244    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:09.753527 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:09.753499    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:09.879864 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:09.879790    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:10.003986 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:10.003846    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:10.330921 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:10.330794    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:10.360543 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:10.360513    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:57:10.361062 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:10.361043    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:57:10.361774 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:10.361757    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 22:57:10.702506 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:10.702471    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:11.001931 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.001839    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:11.143921 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.143880    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:11.157944 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.157910    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"]
Oct 29 22:57:11.163427 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.163399    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.181725 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.181695    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-envoy\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.181867 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.181744    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z274x\" (UniqueName: \"kubernetes.io/projected/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-kube-api-access-z274x\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.181948 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.181906    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istiod-ca-cert\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.182052 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.182019    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-workload-socket\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.182124 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.182091    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-workload-certs\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.182182 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.182166    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-credential-socket\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.182296 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.182279    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-data\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.182333 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.182321    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-podinfo\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.182385 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.182346    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-token\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.184561 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.184542    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-79e6c631-openshift-default-dockercfg-vgjfr\""
Oct 29 22:57:11.196337 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.196311    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"]
Oct 29 22:57:11.275218 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.275135    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:11.283513 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.283481    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istiod-ca-cert\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.283696 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.283547    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-workload-socket\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.283696 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.283568    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-workload-certs\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.283696 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.283609    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-credential-socket\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.283696 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.283647    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-data\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.283696 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.283692    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-podinfo\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.284140 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.283724    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-token\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.284140 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.283829    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-envoy\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.284140 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.283865    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z274x\" (UniqueName: \"kubernetes.io/projected/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-kube-api-access-z274x\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.284299 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.284175    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-workload-socket\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.284299 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.284244    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istiod-ca-cert\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.284407 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.284351    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-workload-certs\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.284650 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.284628    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-credential-socket\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.284743 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.284638    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-data\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.286405 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.286381    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-envoy\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.286787 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.286765    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-podinfo\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.305271 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.305243    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-token\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.305632 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.305610    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z274x\" (UniqueName: \"kubernetes.io/projected/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-kube-api-access-z274x\") pod \"e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") " pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.425216 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.424989    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:11.483552 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.483520    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:11.820939 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.820914    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"]
Oct 29 22:57:11.822723 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:57:11.822696    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7de2d2a_c088_4e76_8a0d_f4e3cb0215ff.slice/crio-3186fa3fddc617452fc877edbc475a63d61f08668289be9d7036479c2e126ad1 WatchSource:0}: Error finding container 3186fa3fddc617452fc877edbc475a63d61f08668289be9d7036479c2e126ad1: Status 404 returned error can't find the container with id 3186fa3fddc617452fc877edbc475a63d61f08668289be9d7036479c2e126ad1
Oct 29 22:57:11.825137 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.825107    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:11.825216 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.825170    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:11.825216 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:11.825200    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:12.299248 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:12.299212    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:12.479849 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:12.479782    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll" event={"ID":"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff","Type":"ContainerStarted","Data":"f19ed61957bd77efa472c6ef591da14dac3b9e8abe6fb8c4e652edcc866c7d4e"}
Oct 29 22:57:12.480020 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:12.479877    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll" event={"ID":"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff","Type":"ContainerStarted","Data":"3186fa3fddc617452fc877edbc475a63d61f08668289be9d7036479c2e126ad1"}
Oct 29 22:57:12.484233 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:12.484208    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:12.491208 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:12.491184    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:12.534774 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:12.534724    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll" podStartSLOduration=1.5347110370000001 podStartE2EDuration="1.534711037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:12.531961922 +0000 UTC m=+835.874910685" watchObservedRunningTime="2026-10-29 22:57:12.534711037 +0000 UTC m=+835.877659782"
Oct 29 22:57:13.495685 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:13.495651    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:13.496876 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:13.496849    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:57:14.318193 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:14.318157    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:15.015651 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:15.015618    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:15.150538 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:15.150496    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:16.287516 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:16.287471    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:16.668021 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:16.667976    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:16.794360 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:16.794331    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:16.838256 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:16.838223    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz"]
Oct 29 22:57:16.843461 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:16.843441    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz"
Oct 29 22:57:16.869284 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:16.868654    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-79e6c631\""
Oct 29 22:57:16.869284 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:16.868834    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-79e6c631-dockercfg-7cbf6\""
Oct 29 22:57:16.874460 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:16.874439    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz"]
Oct 29 22:57:16.982322 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:16.968182    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3982f117-51f0-44b5-aa87-a5a6f3132fab-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz\" (UID: \"3982f117-51f0-44b5-aa87-a5a6f3132fab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz"
Oct 29 22:57:17.087731 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:17.087476    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbhzp\" (UniqueName: \"kubernetes.io/projected/3982f117-51f0-44b5-aa87-a5a6f3132fab-kube-api-access-gbhzp\") pod \"payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz\" (UID: \"3982f117-51f0-44b5-aa87-a5a6f3132fab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz"
Oct 29 22:57:17.090150 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:17.088770    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3982f117-51f0-44b5-aa87-a5a6f3132fab-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz\" (UID: \"3982f117-51f0-44b5-aa87-a5a6f3132fab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz"
Oct 29 22:57:17.090150 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:17.090003    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3982f117-51f0-44b5-aa87-a5a6f3132fab-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz\" (UID: \"3982f117-51f0-44b5-aa87-a5a6f3132fab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz"
Oct 29 22:57:17.190041 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:17.190002    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhzp\" (UniqueName: \"kubernetes.io/projected/3982f117-51f0-44b5-aa87-a5a6f3132fab-kube-api-access-gbhzp\") pod \"payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz\" (UID: \"3982f117-51f0-44b5-aa87-a5a6f3132fab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz"
Oct 29 22:57:17.212781 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:17.212752    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbhzp\" (UniqueName: \"kubernetes.io/projected/3982f117-51f0-44b5-aa87-a5a6f3132fab-kube-api-access-gbhzp\") pod \"payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz\" (UID: \"3982f117-51f0-44b5-aa87-a5a6f3132fab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz"
Oct 29 22:57:17.367999 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:57:17.367953    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:57:17.460614 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:17.460568    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz"
Oct 29 22:57:17.778868 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:17.778842    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz"]
Oct 29 22:57:17.780512 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:57:17.780481    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3982f117_51f0_44b5_aa87_a5a6f3132fab.slice/crio-434c34d76988b6f2ec79ded82ac41f25ccf7847cc4b028d459650aa378ede234 WatchSource:0}: Error finding container 434c34d76988b6f2ec79ded82ac41f25ccf7847cc4b028d459650aa378ede234: Status 404 returned error can't find the container with id 434c34d76988b6f2ec79ded82ac41f25ccf7847cc4b028d459650aa378ede234
Oct 29 22:57:18.277515 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:18.277480    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:18.536943 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:18.536857    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz" event={"ID":"3982f117-51f0-44b5-aa87-a5a6f3132fab","Type":"ContainerStarted","Data":"fb497befbae7bcbfc29c081478d54251e1d8a2433f1deb9acf602901a3ffcf3f"}
Oct 29 22:57:18.536943 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:18.536919    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz" event={"ID":"3982f117-51f0-44b5-aa87-a5a6f3132fab","Type":"ContainerStarted","Data":"434c34d76988b6f2ec79ded82ac41f25ccf7847cc4b028d459650aa378ede234"}
Oct 29 22:57:18.537343 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:18.536949    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz"
Oct 29 22:57:18.576368 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:18.576318    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz" podStartSLOduration=2.5763041920000003 podStartE2EDuration="2.576304192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:18.573122125 +0000 UTC m=+841.916070862" watchObservedRunningTime="2026-10-29 22:57:18.576304192 +0000 UTC m=+841.919252937"
Oct 29 22:57:19.295879 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:19.295844    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:19.620597 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:19.620568    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:20.194926 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:20.194885    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:21.835516 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:21.835470    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"]
Oct 29 22:57:21.839982 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:21.839918    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:21.874216 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:21.874179    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"]
Oct 29 22:57:21.913903 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:21.913851    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:21.948951 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:21.948915    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:21.949154 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:21.948962    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhjnw\" (UniqueName: \"kubernetes.io/projected/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-kube-api-access-bhjnw\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:21.949154 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:21.948988    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:21.949154 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:21.949143    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:21.949492 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:21.949471    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:21.949637 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:21.949545    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:21.949746 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:21.949719    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:21.949916 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:21.949889    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:21.949981 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:21.949959    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.051369 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.051340    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.051543 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.051381    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.051543 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.051442    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.051543 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.051475    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.051659 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.051597    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.051711 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.051667    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhjnw\" (UniqueName: \"kubernetes.io/projected/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-kube-api-access-bhjnw\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.051763 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.051707    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.051831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.051756    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.051895 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.051827    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.051895 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.051867    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.052145 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.052120    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.052244 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.052212    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.052384 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.052365    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.052466 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.052377    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.054250 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.054225    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.054687 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.054670    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.073131 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.073099    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhjnw\" (UniqueName: \"kubernetes.io/projected/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-kube-api-access-bhjnw\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.073131 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.073126    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.156161 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.156096    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:22.482606 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.482574    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"]
Oct 29 22:57:22.484103 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:57:22.484074    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb457bfe9_1262_4ba5_bd19_fbdfd0e14310.slice/crio-492124a193f64ba151284dba83eaf2be893c6789b943bfac4446e6d1a599e3fd WatchSource:0}: Error finding container 492124a193f64ba151284dba83eaf2be893c6789b943bfac4446e6d1a599e3fd: Status 404 returned error can't find the container with id 492124a193f64ba151284dba83eaf2be893c6789b943bfac4446e6d1a599e3fd
Oct 29 22:57:22.487231 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.487190    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:22.487322 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.487280    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:22.487373 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.487318    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:22.575694 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.575657    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm" event={"ID":"b457bfe9-1262-4ba5-bd19-fbdfd0e14310","Type":"ContainerStarted","Data":"79aa1c26da7fd4420274106f637e173b8df7b354e5eb44e4389f34fe624abdb7"}
Oct 29 22:57:22.575835 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.575717    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm" event={"ID":"b457bfe9-1262-4ba5-bd19-fbdfd0e14310","Type":"ContainerStarted","Data":"492124a193f64ba151284dba83eaf2be893c6789b943bfac4446e6d1a599e3fd"}
Oct 29 22:57:22.616646 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:22.616592    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm" podStartSLOduration=1.6165782370000001 podStartE2EDuration="1.616578237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:22.616163932 +0000 UTC m=+845.959112679" watchObservedRunningTime="2026-10-29 22:57:22.616578237 +0000 UTC m=+845.959527051"
Oct 29 22:57:23.157045 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.157010    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:23.158313 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.158290    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:57:23.158385 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.158337    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm" podUID="b457bfe9-1262-4ba5-bd19-fbdfd0e14310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 29 22:57:23.290096 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.290055    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:23.304711 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.304679    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"]
Oct 29 22:57:23.309074 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.309052    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.338061 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.338030    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"]
Oct 29 22:57:23.388454 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.388406    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.388669 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.388474    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.388669 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.388616    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.388794 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.388711    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.388794 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.388752    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjrxv\" (UniqueName: \"kubernetes.io/projected/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-kube-api-access-wjrxv\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.388954 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.388932    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.388991 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.388973    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.389029 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.389014    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.389145 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.389128    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.490763 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.490681    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.490763 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.490725    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.491039 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.490766    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.491039 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.490922    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.491039 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.490969    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.491039 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.491015    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.491223 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.491078    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.491223 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.491156    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.491223 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.491193    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjrxv\" (UniqueName: \"kubernetes.io/projected/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-kube-api-access-wjrxv\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.491369 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.491226    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.491768 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.491742    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.492037 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.492001    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.492175 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.492144    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.492396 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.492379    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.494050 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.494025    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.494303 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.494284    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.512916 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.512889    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.513187 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.513164    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjrxv\" (UniqueName: \"kubernetes.io/projected/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-kube-api-access-wjrxv\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.554518 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.554490    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:23.626249 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.626217    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:23.995467 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.995443    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"]
Oct 29 22:57:23.996776 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:57:23.996751    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cae6a36_4e41_4fda_a0e5_ceaa7a957cb9.slice/crio-be6d140dee660ece155bbbfc14587a93f788aeb1ede8a2c5388b046d65e1a833 WatchSource:0}: Error finding container be6d140dee660ece155bbbfc14587a93f788aeb1ede8a2c5388b046d65e1a833: Status 404 returned error can't find the container with id be6d140dee660ece155bbbfc14587a93f788aeb1ede8a2c5388b046d65e1a833
Oct 29 22:57:24.000029 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:23.999984    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:24.000160 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:24.000139    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:24.000246 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:24.000229    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:24.174823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:24.156536    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:57:24.174823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:24.156596    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm" podUID="b457bfe9-1262-4ba5-bd19-fbdfd0e14310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 29 22:57:24.326737 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:24.326645    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:24.607883 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:24.606995    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm" event={"ID":"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9","Type":"ContainerStarted","Data":"d35dfdc14fc08f53212ad77ef76163513c0108fd2e4ff08315b0b13d2bc643d3"}
Oct 29 22:57:24.607883 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:24.607093    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm" event={"ID":"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9","Type":"ContainerStarted","Data":"be6d140dee660ece155bbbfc14587a93f788aeb1ede8a2c5388b046d65e1a833"}
Oct 29 22:57:24.627051 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:24.627017    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:24.679827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:24.679673    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm" podStartSLOduration=1.679653303 podStartE2EDuration="1.679653303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:24.658116305 +0000 UTC m=+848.001065050" watchObservedRunningTime="2026-10-29 22:57:24.679653303 +0000 UTC m=+848.022602051"
Oct 29 22:57:25.157199 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:25.157162    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:57:25.157375 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:25.157218    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm" podUID="b457bfe9-1262-4ba5-bd19-fbdfd0e14310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 29 22:57:25.297825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:25.295099    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:25.627911 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:25.627880    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:57:25.628076 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:25.627929    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm" podUID="9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 29 22:57:26.156900 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:26.156864    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:57:26.157079 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:26.156922    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm" podUID="b457bfe9-1262-4ba5-bd19-fbdfd0e14310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 29 22:57:26.627226 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:26.627196    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:57:26.627622 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:26.627245    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm" podUID="9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 29 22:57:27.091104 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:27.091010    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:27.157105 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:27.157070    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:57:27.157287 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:27.157129    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm" podUID="b457bfe9-1262-4ba5-bd19-fbdfd0e14310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 29 22:57:27.626576 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:27.626533    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:57:27.626788 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:27.626592    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm" podUID="9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 29 22:57:28.156509 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.156470    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:57:28.156937 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.156525    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm" podUID="b457bfe9-1262-4ba5-bd19-fbdfd0e14310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 29 22:57:28.254741 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.254696    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"]
Oct 29 22:57:28.271094 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.270282    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.325835 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.325782    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"]
Oct 29 22:57:28.412873 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.412742    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.412873 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.412830    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0748e57c-035e-47c3-a76f-bebe19de3139-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.413119 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.412890    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.413119 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.412951    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.413119 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.412974    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.413119 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.413016    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0748e57c-035e-47c3-a76f-bebe19de3139-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.413119 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.413102    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.413382 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.413141    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0748e57c-035e-47c3-a76f-bebe19de3139-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.413382 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.413169    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swdkh\" (UniqueName: \"kubernetes.io/projected/0748e57c-035e-47c3-a76f-bebe19de3139-kube-api-access-swdkh\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.514352 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.514312    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.514602 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.514373    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0748e57c-035e-47c3-a76f-bebe19de3139-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.514602 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.514428    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.514602 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.514485    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.514602 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.514510    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.514602 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.514552    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0748e57c-035e-47c3-a76f-bebe19de3139-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.514889 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.514687    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.514889 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.514730    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0748e57c-035e-47c3-a76f-bebe19de3139-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.514889 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.514740    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-workload-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.514889 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.514760    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swdkh\" (UniqueName: \"kubernetes.io/projected/0748e57c-035e-47c3-a76f-bebe19de3139-kube-api-access-swdkh\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.515312 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.515283    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-istio-data\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.515734 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.515707    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0748e57c-035e-47c3-a76f-bebe19de3139-istiod-ca-cert\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.516072 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.516041    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-workload-certs\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.516211 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.516189    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-credential-socket\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.519005 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.518981    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-istio-envoy\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.519724 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.519703    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0748e57c-035e-47c3-a76f-bebe19de3139-istio-podinfo\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.536632 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.536599    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swdkh\" (UniqueName: \"kubernetes.io/projected/0748e57c-035e-47c3-a76f-bebe19de3139-kube-api-access-swdkh\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.537069 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.537045    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0748e57c-035e-47c3-a76f-bebe19de3139-istio-token\") pod \"e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") " pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.625961 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.625921    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:28.627184 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.627149    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:57:28.627335 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.627217    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm" podUID="9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 29 22:57:28.948624 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:28.948582    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:28.999453 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:57:28.999280    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0748e57c_035e_47c3_a76f_bebe19de3139.slice/crio-5ba98790671be4401efe75773d19c786b6dfa59a6c93d1cf23034635b66503fd WatchSource:0}: Error finding container 5ba98790671be4401efe75773d19c786b6dfa59a6c93d1cf23034635b66503fd: Status 404 returned error can't find the container with id 5ba98790671be4401efe75773d19c786b6dfa59a6c93d1cf23034635b66503fd
Oct 29 22:57:29.001603 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:29.001576    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"]
Oct 29 22:57:29.002146 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:29.002117    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:29.002255 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:29.002181    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:29.002255 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:29.002209    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:29.084063 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:29.080436    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:29.207996 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:29.203519    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:57:29.207996 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:29.203573    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm" podUID="b457bfe9-1262-4ba5-bd19-fbdfd0e14310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 29 22:57:29.226888 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:29.219938    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:29.547420 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:29.547385    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz"
Oct 29 22:57:29.627825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:29.627528    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:57:29.627825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:29.627604    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm" podUID="9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 29 22:57:29.652514 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:29.652470    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc" event={"ID":"0748e57c-035e-47c3-a76f-bebe19de3139","Type":"ContainerStarted","Data":"2db79b7a37f4baf62119981bb9db223b7283b031802cc077d0929536a08427f5"}
Oct 29 22:57:29.652772 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:29.652540    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc" event={"ID":"0748e57c-035e-47c3-a76f-bebe19de3139","Type":"ContainerStarted","Data":"5ba98790671be4401efe75773d19c786b6dfa59a6c93d1cf23034635b66503fd"}
Oct 29 22:57:30.437824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.435197    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:30.525838 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.525449    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc" podStartSLOduration=2.525428272 podStartE2EDuration="2.525428272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:29.715274072 +0000 UTC m=+853.058222818" watchObservedRunningTime="2026-10-29 22:57:30.525428272 +0000 UTC m=+853.868377017"
Oct 29 22:57:30.531825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.529940    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"]
Oct 29 22:57:30.536071 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.536047    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.621828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.608864    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"]
Oct 29 22:57:30.629497 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.629460    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:30.630046 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.629945    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 29 22:57:30.630046 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.629996    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm" podUID="9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 29 22:57:30.635642 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.635611    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:57:30.635825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.635663    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc" podUID="0748e57c-035e-47c3-a76f-bebe19de3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 29 22:57:30.691880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.666510    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.691880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.666562    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.691880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.666600    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.691880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.666705    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cf99859-eb40-4c1f-a856-c92a700e01eb-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.691880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.666733    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9j5w\" (UniqueName: \"kubernetes.io/projected/9cf99859-eb40-4c1f-a856-c92a700e01eb-kube-api-access-r9j5w\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.691880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.666761    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.691880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.666822    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.691880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.666922    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.691880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.667004    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.691880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.674117    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:30.691880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.689636    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:30.708852 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.705202    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 22:57:30.791827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.787725    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.793578 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.792246    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cf99859-eb40-4c1f-a856-c92a700e01eb-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.793829 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.793788    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9j5w\" (UniqueName: \"kubernetes.io/projected/9cf99859-eb40-4c1f-a856-c92a700e01eb-kube-api-access-r9j5w\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.793980 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.793961    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.794168 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.794151    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.794452 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.794430    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.794943 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.794906    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.795217 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.795085    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.796339 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.795214    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.796339 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.795262    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.796339 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.795891    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.797024 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.796997    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.797275 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.797250    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.798397 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.798373    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cf99859-eb40-4c1f-a856-c92a700e01eb-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.814286 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.811082    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.822342 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.822015    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9j5w\" (UniqueName: \"kubernetes.io/projected/9cf99859-eb40-4c1f-a856-c92a700e01eb-kube-api-access-r9j5w\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.825523 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.825442    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.835073 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.835034    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.858057 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.857999    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:30.967844 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:30.966412    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"]
Oct 29 22:57:31.038711 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.038675    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"]
Oct 29 22:57:31.039754 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.039726    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.092841 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.090609    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"]
Oct 29 22:57:31.092841 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.090664    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"]
Oct 29 22:57:31.092841 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.090773    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.121821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.121093    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.121821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.121180    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plkrv\" (UniqueName: \"kubernetes.io/projected/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-kube-api-access-plkrv\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.121821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.121219    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.121821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.121262    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.121821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.121499    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.121821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.121576    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.121821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.121624    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.121821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.121682    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.121821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.121711    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.223994 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.223861    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.223994 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.223936    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.224248 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.223998    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.224248 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.224080    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.224248 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.224139    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.224248 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.224181    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7007018-33a5-4c01-b518-5a9f17273efb-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.224248 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.224229    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.224514 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.224303    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.224514 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.224331    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.224720 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.224697    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.224827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.224741    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.225008 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.224982    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.225111 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.225086    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.225175 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.224373    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.226301 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.225378    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.226301 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.225459    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7007018-33a5-4c01-b518-5a9f17273efb-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.226301 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.225532    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.226301 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.225756    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7007018-33a5-4c01-b518-5a9f17273efb-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.226301 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.225782    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.226301 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.225841    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.226301 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.225880    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crz95\" (UniqueName: \"kubernetes.io/projected/b7007018-33a5-4c01-b518-5a9f17273efb-kube-api-access-crz95\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.226301 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.225934    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.226301 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.225982    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plkrv\" (UniqueName: \"kubernetes.io/projected/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-kube-api-access-plkrv\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.229039 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.229009    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.229682 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.229659    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.246882 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.246853    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.247608 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.247560    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plkrv\" (UniqueName: \"kubernetes.io/projected/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-kube-api-access-plkrv\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.299997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.299466    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"]
Oct 29 22:57:31.304566 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.304525    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:31.304671 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.304615    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:31.304671 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.304663    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:31.327145 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.327102    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.327454 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.327297    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.327454 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.327365    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7007018-33a5-4c01-b518-5a9f17273efb-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.327454 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.327406    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.327454 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.327405    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-credential-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.327454 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.327454    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7007018-33a5-4c01-b518-5a9f17273efb-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.327644 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.327501    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.327677 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.327647    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-workload-socket\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.327677 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.327663    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7007018-33a5-4c01-b518-5a9f17273efb-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.327755 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.327712    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.327823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.327752    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crz95\" (UniqueName: \"kubernetes.io/projected/b7007018-33a5-4c01-b518-5a9f17273efb-kube-api-access-crz95\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.328240 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.328195    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-workload-certs\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.328240 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.328231    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-istio-data\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.328396 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.328233    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7007018-33a5-4c01-b518-5a9f17273efb-istiod-ca-cert\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.329862 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.329768    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-istio-envoy\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.330896 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.330873    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7007018-33a5-4c01-b518-5a9f17273efb-istio-podinfo\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.355693 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.355647    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crz95\" (UniqueName: \"kubernetes.io/projected/b7007018-33a5-4c01-b518-5a9f17273efb-kube-api-access-crz95\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.355851 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.355667    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7007018-33a5-4c01-b518-5a9f17273efb-istio-token\") pod \"e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") " pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.435027 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.434769    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:31.455265 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.455232    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:31.628872 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.628378    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:57:31.628872 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.628441    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc" podUID="0748e57c-035e-47c3-a76f-bebe19de3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 29 22:57:31.663336 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.655976    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:31.765766 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.764845    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" event={"ID":"9cf99859-eb40-4c1f-a856-c92a700e01eb","Type":"ContainerStarted","Data":"bb8993ed4e82ae873ea239d84ba5b436c9339c3f3a3ba45237acb9f59d03aac2"}
Oct 29 22:57:31.765766 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.764912    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" event={"ID":"9cf99859-eb40-4c1f-a856-c92a700e01eb","Type":"ContainerStarted","Data":"1c7716a95424b1cc6f1f49db365085b16f978e2de895e1dbab6f276dc55cab14"}
Oct 29 22:57:31.907009 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.906971    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:31.954530 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.954423    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:31.955263 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.955115    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:31.956129 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:31.956106    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 22:57:32.019606 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.018210    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podStartSLOduration=2.018190327 podStartE2EDuration="2.018190327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:31.860623113 +0000 UTC m=+855.203571860" watchObservedRunningTime="2026-10-29 22:57:32.018190327 +0000 UTC m=+855.361139074"
Oct 29 22:57:32.125961 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.124370    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"]
Oct 29 22:57:32.131415 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.131366    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:32.131514 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.131450    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:32.131574 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.131515    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:32.137459 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.137430    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:32.137642 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.137622    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:32.137767 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.137752    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:32.140194 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.140090    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"]
Oct 29 22:57:32.630467 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.629979    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:57:32.630467 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.630042    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc" podUID="0748e57c-035e-47c3-a76f-bebe19de3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 29 22:57:32.835636 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.835589    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" event={"ID":"91ad351f-5ccd-4777-8abd-7a02bdcb23eb","Type":"ContainerStarted","Data":"4ffee01fbcaecb0deb64c6dc66834dd3d42d58dfd3da9b02bce5ebe092817eb7"}
Oct 29 22:57:32.835826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.835671    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" event={"ID":"91ad351f-5ccd-4777-8abd-7a02bdcb23eb","Type":"ContainerStarted","Data":"26f983fc376a6e45e6f4f15fa7141e4cfc6ee12846ebb3ce3d8aa2764dfbea45"}
Oct 29 22:57:32.843822 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.843765    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" event={"ID":"b7007018-33a5-4c01-b518-5a9f17273efb","Type":"ContainerStarted","Data":"318bc9a014b14fe87878294d7866c39afd13e918b872df48056271089c144260"}
Oct 29 22:57:32.844116 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.844089    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" event={"ID":"b7007018-33a5-4c01-b518-5a9f17273efb","Type":"ContainerStarted","Data":"ee025b793e3011c98fb1abae3b3e05cfaadb6788c28fdce75f32c5b8f1ed9d59"}
Oct 29 22:57:32.900883 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.883953    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:57:32.900883 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.884024    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:57:32.918826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.918276    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" podStartSLOduration=2.918255334 podStartE2EDuration="2.918255334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:32.884669434 +0000 UTC m=+856.227618272" watchObservedRunningTime="2026-10-29 22:57:32.918255334 +0000 UTC m=+856.261204081"
Oct 29 22:57:32.939852 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:32.939715    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" podStartSLOduration=2.939704709 podStartE2EDuration="2.939704709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:32.939528209 +0000 UTC m=+856.282476953" watchObservedRunningTime="2026-10-29 22:57:32.939704709 +0000 UTC m=+856.282653453"
Oct 29 22:57:33.440137 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:33.437686    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:57:33.440137 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:33.437768    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:57:33.440137 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:33.438093    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:33.460290 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:33.460159    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:33.461506 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:33.461477    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:57:33.461676 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:33.461651    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:57:33.635829 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:33.621069    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:33.635829 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:33.634469    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:57:33.635829 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:33.634523    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc" podUID="0748e57c-035e-47c3-a76f-bebe19de3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 29 22:57:33.879824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:33.871547    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:57:33.879824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:33.871604    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:57:34.436083 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:34.435620    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:57:34.436083 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:34.435684    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:57:34.455928 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:34.455792    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:57:34.455928 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:34.455867    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:57:34.497816 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:34.497334    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:34.553912 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:34.553092    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:34.628281 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:34.628242    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:57:34.628557 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:34.628529    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc" podUID="0748e57c-035e-47c3-a76f-bebe19de3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 29 22:57:34.728652 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:34.728547    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:34.862826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:34.862354    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:57:34.862826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:34.862421    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:57:35.036912 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:35.028754    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:57:35.123187 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:35.087917    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:35.160827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:35.157074    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:35.247028 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:35.246966    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 29 22:57:35.492825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:35.442247    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:57:35.492825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:35.442312    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:57:35.494521 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:35.494365    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:57:35.518825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:35.507897    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:35.518825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:35.508429    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:57:35.566827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:35.566408    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:57:35.590828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:35.567123    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:57:35.740337 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:35.740053    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:35.766790 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:35.518977    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:57:35.812826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:35.811910    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:35.974539 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:35.959045    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:57:35.974539 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:35.969673    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:57:35.974539 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:35.969728    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc" podUID="0748e57c-035e-47c3-a76f-bebe19de3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 29 22:57:36.030908 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.030819    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:57:36.032139 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.032109    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded"
Oct 29 22:57:36.055820 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.032357    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:57:36.082090 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.081982    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:57:36.082399 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.082338    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:57:36.104065 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.103974    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:36.149519 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.149487    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:57:36.293684 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.293554    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded"
Oct 29 22:57:36.293991 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.293965    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 29 22:57:36.299951 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.299922    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:57:36.355191 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.355141    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc" podUID="c4034d76-e0f2-4674-928d-657820813e88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:36.579884 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.500102    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:57:36.579884 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.574120    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:57:36.579884 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.513825    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:36.579884 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.501024    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:36.579884 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.524081    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:57:36.579884 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.575450    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:57:36.579884 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.573931    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:36.664825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.649069    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:57:36.664825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.649130    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc" podUID="0748e57c-035e-47c3-a76f-bebe19de3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 29 22:57:36.754504 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.754452    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:36.833885 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.817026    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:36.833885 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.817130    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:36.915895 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.896292    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:57:36.915895 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:36.896362    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:57:37.050878 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:37.040264    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:57:37.298669 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:37.255351    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:37.298669 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:37.294732    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:37.345822 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:37.332410    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:57:37.461825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:37.443880    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:57:37.461825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:37.443939    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:57:37.530826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:37.529818    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:57:37.530826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:37.529878    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:57:37.589109 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:37.589006    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:57:37.589698 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:37.589328    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:37.648692 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:37.648646    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 29 22:57:37.648991 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:37.648960    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc" podUID="0748e57c-035e-47c3-a76f-bebe19de3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 29 22:57:37.789945 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:37.789897    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:37.826696 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:37.826302    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:37.883022 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:37.882988    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:57:37.911827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:37.886879    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:57:38.123828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:38.110955    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:38.423827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:38.385329    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:38.423827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:38.385689    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 29 22:57:38.453843 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:38.449919    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:57:38.453843 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:38.449988    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:57:38.516854 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:38.490429    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:57:38.516854 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:38.490486    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:57:38.877889 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:38.843126    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:39.211878 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:39.210151    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:57:39.211878 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:39.210221    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:57:39.211878 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:39.116135    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:57:39.211878 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:38.939982    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:39.245157 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:39.225858    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded"
Oct 29 22:57:39.276048 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:38.939940    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:57:39.417868 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:39.401666    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:39.520062 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:39.519958    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:57:39.672375 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:39.672000    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:40.071827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:39.998282    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:57:40.092295 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.092253    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:57:40.106849 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.035301    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:57:40.143828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.128452    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:57:40.143828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.035359    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:57:40.143828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.132167    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:57:40.145382 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.136134    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:40.227122 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.227074    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:40.397826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.247831    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:40.397826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.323041    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:40.402326 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.401951    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 29 22:57:40.411987 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.411949    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:40.412229 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.412199    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc" podUID="0748e57c-035e-47c3-a76f-bebe19de3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:40.468825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.459555    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:57:40.468825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.459623    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:57:40.468825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.459713    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:57:40.468825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.459745    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:57:40.527826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.523043    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:40.907903 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.718988    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 29 22:57:40.907903 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.878388    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:40.952865 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.932142    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:57:40.952865 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.932201    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:57:40.998499 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:40.998457    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.218699 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.218596    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 29 22:57:41.373961 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.373905    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.510969 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.380040    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:57:41.511449 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.380684    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:57:41.511854 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.409930    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.512235 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.410623    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:41.512423 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.438590    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:57:41.512559 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.512533    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:57:41.512698 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.444897    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:41.512829 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.512790    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc" podUID="0748e57c-035e-47c3-a76f-bebe19de3139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.513127 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.464572    5428 patch_prober.go:29] interesting pod/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:41.533281 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.533248    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr" podUID="53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.890954 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.533177    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.890954 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.468290    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:57:41.890954 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.886884    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:57:41.890954 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.878648    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:57:41.890954 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.886949    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:57:41.890954 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.857984    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.923826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.922923    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:41.946827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:41.924502    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:42.044828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:42.020084    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 22:57:42.405956 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:42.258130    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 29 22:57:42.479829 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:42.479467    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:57:42.479829 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:42.479533    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:57:42.505013 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:42.479986    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:57:42.505013 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:42.480045    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:57:42.529827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:42.517684    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:42.529827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:42.518197    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:57:42.538942 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:42.538892    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:57:42.546826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:42.539532    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:42.761888 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:42.746094    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:42.875855 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:42.872788    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:57:42.875855 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:42.872880    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:57:42.891854 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:42.889139    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:42.892091 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:42.892059    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:43.186043 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:43.184935    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-98p7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:43.186043 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:43.185008    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b" podUID="bf52a0e8-c739-47f8-9676-c5261e8d0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:43.350828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:43.325329    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:43.449888 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:43.449040    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-d9nr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:43.449888 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:43.449112    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2" podUID="9d3977ef-edc8-47f1-9779-c0f257e0e475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:43.449888 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:43.449553    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:57:43.449888 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:43.449603    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:57:43.467469 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:43.457356    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:57:43.467469 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:43.457418    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:57:43.539936 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:43.539885    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:43.541036 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:43.541001    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:57:43.723099 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:43.550064    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:57:43.745264 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:43.594958    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:43.844038 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:43.843978    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:43.844357 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:43.844304    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264" podUID="f56ff730-ff0f-47f0-be77-3bbde2f85301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:43.882415 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:43.882377    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:57:44.135903 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:44.114880    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:57:44.135903 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:43.915885    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:44.135903 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:43.889516    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:44.252828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:44.227859    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded"
Oct 29 22:57:44.354981 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:44.354937    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 29 22:57:44.506927 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:44.502855    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:57:44.506927 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:44.502929    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:57:44.506927 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:44.503435    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:57:44.506927 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:44.503488    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:57:44.560837 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:44.540944    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:44.611338 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:44.611288    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:44.738949 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:44.738902    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:44.754386 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:44.754346    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:45.046537 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.046039    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:57:45.046537 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.046118    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 29 22:57:45.117361 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.116326    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:45.145837 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.124626    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 29 22:57:45.145837 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.144182    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:45.443931 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.439276    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:57:45.443931 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.439337    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:57:45.463506 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.460988    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:57:45.463506 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.461048    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:57:45.547108 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.547059    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:45.564709 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.563930    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:45.639283 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.639237    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:57:45.679261 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.675264    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"]
Oct 29 22:57:45.750266 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.749090    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:45.846826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.820083    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:45.846826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.820921    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.849979 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.849203    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.849979 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.849263    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kct9b\" (UniqueName: \"kubernetes.io/projected/7c5d481f-83bd-48f3-8945-cd88d304e8d2-kube-api-access-kct9b\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.849979 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.849370    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.849979 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.849452    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.849979 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.849568    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.849979 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.849651    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.849979 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.849680    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.849979 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.849707    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.849979 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.849736    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.853821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.803178    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"]
Oct 29 22:57:45.933876 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.924227    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:45.962834 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.951597    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.962834 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.951658    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kct9b\" (UniqueName: \"kubernetes.io/projected/7c5d481f-83bd-48f3-8945-cd88d304e8d2-kube-api-access-kct9b\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.962834 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.951762    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.962834 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.951859    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.962834 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.951971    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.962834 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.952049    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.962834 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.952083    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.962834 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.952121    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.962834 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.952148    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.962834 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.954176    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.962834 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.954753    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.964234 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.964210    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.964664 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.964643    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:45.965408 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:45.965388    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:46.007638 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.007554    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kct9b\" (UniqueName: \"kubernetes.io/projected/7c5d481f-83bd-48f3-8945-cd88d304e8d2-kube-api-access-kct9b\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:46.020824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.016343    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:46.020824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.016436    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:46.038830 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.036620    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:46.052830 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.051672    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:46.137828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.125459    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:46.137828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.125860    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:46.196123 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.185934    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:46.218778 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.218467    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:46.422872 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.416103    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:57:46.437833 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.436549    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:46.460832 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.459906    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:46.476424 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.461308    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:57:46.476424 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.461361    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:57:46.477487 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.477352    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:57:46.477487 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.477427    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:57:46.553886 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.545515    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:46.554560 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.548278    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 29 22:57:46.569873 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.569837    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:46.703591 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.689929    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:46.740823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.740602    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:46.785473 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.759462    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:57:46.960163 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.960124    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:46.960456 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:46.960421    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:47.149388 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:47.125835    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:47.149388 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:47.126409    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:57:47.504214 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:47.495863    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:57:47.504214 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:47.495942    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 29 22:57:47.504214 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:47.497866    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:57:47.504214 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:47.497915    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 29 22:57:47.580071 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:47.578461    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:47.580071 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:47.579536    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 29 22:57:47.652341 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:47.604913    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:47.709370 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:47.701654    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:47.797999 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:47.797955    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:47.859297 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:47.859258    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:47.909047 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:47.881218    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:47.909047 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:47.881318    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:47.909047 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:47.904982    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:47.983277 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:57:47.977328    5428 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.19s"
Oct 29 22:57:47.987333 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:47.987277    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:57:48.021914 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:48.021877    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:48.022164 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:48.022134    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:48.029364 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:48.029334    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"]
Oct 29 22:57:48.132817 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:48.129163    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:48.143854 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:48.141276    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:48.262980 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:57:48.262939    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:57:48.273971 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:48.273942    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:57:48.291589 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:48.291517    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:57:48.934627 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:48.934591    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:49.017342 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.016920    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 29 22:57:49.100213 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.099030    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:49.100213 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.099092    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:49.131094 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.130951    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:49.142607 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.142556    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:57:49.281365 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.281261    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:49.281675 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.281658    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:49.281909 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.281896    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:49.282110 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.282098    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:57:49.282248 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.282238    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:57:49.282377 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.282366    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:57:49.383167 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.383138    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 22:57:49.383508 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.383487    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 22:57:49.406521 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.387765    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 22:57:49.417503 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.417467    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n" event={"ID":"7c5d481f-83bd-48f3-8945-cd88d304e8d2","Type":"ContainerStarted","Data":"6a95b91343d39a5bc231348e910b4c2bdcf3cf1e8aa780addaf1ed4594ab4c47"}
Oct 29 22:57:49.417999 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.417958    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n" event={"ID":"7c5d481f-83bd-48f3-8945-cd88d304e8d2","Type":"ContainerStarted","Data":"b730d96884438d517ac12d8d01866f4cab74547f0c1849a4d6cd369a1c4a92aa"}
Oct 29 22:57:49.657136 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.657100    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:49.856866 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.855671    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:49.917909 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.917772    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n" podStartSLOduration=4.917754027 podStartE2EDuration="4.917754027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:49.805133304 +0000 UTC m=+873.148082050" watchObservedRunningTime="2026-10-29 22:57:49.917754027 +0000 UTC m=+873.260702772"
Oct 29 22:57:49.956376 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:49.956326    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"]
Oct 29 22:57:50.040426 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.040392    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.119825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.115390    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.119825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.115457    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.119825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.115593    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.119825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.115648    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e415acf5-67c8-47c1-aa03-df58dca41b3a-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.119825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.115695    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.119825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.115778    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.151727 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.150568    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.151727 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.151307    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.151727 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.151447    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwdnz\" (UniqueName: \"kubernetes.io/projected/e415acf5-67c8-47c1-aa03-df58dca41b3a-kube-api-access-zwdnz\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.201198 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.201065    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"]
Oct 29 22:57:50.217833 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.207097    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.220229 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.220202    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:50.220987 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.220962    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:57:50.221158 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.221132    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n" podUID="7c5d481f-83bd-48f3-8945-cd88d304e8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 29 22:57:50.322825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.299829    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.322825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.299907    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.322825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.300060    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.322825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.300115    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e415acf5-67c8-47c1-aa03-df58dca41b3a-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.322825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.300168    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.322825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.300255    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.322825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.300290    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.322825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.300316    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.322825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.300348    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwdnz\" (UniqueName: \"kubernetes.io/projected/e415acf5-67c8-47c1-aa03-df58dca41b3a-kube-api-access-zwdnz\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.356838 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.348011    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.358191 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.351040    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.358355 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.351353    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.358445 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.352260    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e415acf5-67c8-47c1-aa03-df58dca41b3a-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.358519 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.352568    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.367086 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.367062    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"]
Oct 29 22:57:50.415555 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.415514    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20a2142-6441-4846-8855-b4595d2fe746-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.415965 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.415938    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20a2142-6441-4846-8855-b4595d2fe746-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.416187 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.416167    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.416353 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.416334    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.416515 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.416498    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.416644 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.416629    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.416826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.416792    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20a2142-6441-4846-8855-b4595d2fe746-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.417030 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.417010    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.417157 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.417140    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk52j\" (UniqueName: \"kubernetes.io/projected/b20a2142-6441-4846-8855-b4595d2fe746-kube-api-access-pk52j\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.421439 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.421410    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.452885 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.434022    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.453147 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.436568    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwdnz\" (UniqueName: \"kubernetes.io/projected/e415acf5-67c8-47c1-aa03-df58dca41b3a-kube-api-access-zwdnz\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.532826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.526119    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.532826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.526173    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk52j\" (UniqueName: \"kubernetes.io/projected/b20a2142-6441-4846-8855-b4595d2fe746-kube-api-access-pk52j\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.532826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.526236    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20a2142-6441-4846-8855-b4595d2fe746-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.532826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.526361    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20a2142-6441-4846-8855-b4595d2fe746-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.532826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.526457    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.532826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.526511    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.532826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.526565    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.532826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.526602    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.532826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.526675    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20a2142-6441-4846-8855-b4595d2fe746-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.532826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.531438    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.533527 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.532994    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20a2142-6441-4846-8855-b4595d2fe746-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.533527 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.533325    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.533682 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.533648    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.537388 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.535853    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.547867 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.547787    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.558639 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.558607    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20a2142-6441-4846-8855-b4595d2fe746-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.622845 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.618403    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.670847 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.652619    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk52j\" (UniqueName: \"kubernetes.io/projected/b20a2142-6441-4846-8855-b4595d2fe746-kube-api-access-pk52j\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.688640 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.688603    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20a2142-6441-4846-8855-b4595d2fe746-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:50.695823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.694575    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:50.737976 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.722861    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"]
Oct 29 22:57:50.804826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:50.803824    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:51.425913 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:51.372710    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:57:51.425913 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:51.372769    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n" podUID="7c5d481f-83bd-48f3-8945-cd88d304e8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 29 22:57:51.640471 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:51.639850    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"]
Oct 29 22:57:51.654682 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:51.654653    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"]
Oct 29 22:57:51.655938 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:51.655878    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:51.656556 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:51.655957    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:51.656556 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:51.655998    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:51.693161 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:57:51.692305    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode415acf5_67c8_47c1_aa03_df58dca41b3a.slice/crio-283f87cd8742df6518d4c7df7b72907d6f411d3bf17bd064133bc41926959173 WatchSource:0}: Error finding container 283f87cd8742df6518d4c7df7b72907d6f411d3bf17bd064133bc41926959173: Status 404 returned error can't find the container with id 283f87cd8742df6518d4c7df7b72907d6f411d3bf17bd064133bc41926959173
Oct 29 22:57:51.696347 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:51.696275    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:51.697331 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:51.696505    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:51.697331 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:51.696568    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:57:52.040284 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:52.039104    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:52.195852 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:52.194718    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:57:52.195852 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:52.194785    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n" podUID="7c5d481f-83bd-48f3-8945-cd88d304e8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 29 22:57:52.598867 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:52.595501    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm" event={"ID":"e415acf5-67c8-47c1-aa03-df58dca41b3a","Type":"ContainerStarted","Data":"6f4f37945e2bebf1405ffe03edddc744f8e15819e70efe15093d46c7b957cb08"}
Oct 29 22:57:52.598867 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:52.595580    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm" event={"ID":"e415acf5-67c8-47c1-aa03-df58dca41b3a","Type":"ContainerStarted","Data":"283f87cd8742df6518d4c7df7b72907d6f411d3bf17bd064133bc41926959173"}
Oct 29 22:57:52.598867 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:52.598046    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z" event={"ID":"b20a2142-6441-4846-8855-b4595d2fe746","Type":"ContainerStarted","Data":"79a1b80d99a5ad05884bf5ec2a2cfab7b78c1b200ff75974faad6081ff247e7f"}
Oct 29 22:57:52.598867 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:52.598106    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z" event={"ID":"b20a2142-6441-4846-8855-b4595d2fe746","Type":"ContainerStarted","Data":"83c47e091796ea8eef41183a1775703b3d4fe837ef195a5153c47cbfd785d6f8"}
Oct 29 22:57:52.652130 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:52.652071    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm" podStartSLOduration=3.65205781 podStartE2EDuration="3.65205781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:52.651032603 +0000 UTC m=+875.993981349" watchObservedRunningTime="2026-10-29 22:57:52.65205781 +0000 UTC m=+875.995006555"
Oct 29 22:57:52.712609 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:52.712554    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z" podStartSLOduration=3.712541104 podStartE2EDuration="3.712541104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:52.708199807 +0000 UTC m=+876.051148570" watchObservedRunningTime="2026-10-29 22:57:52.712541104 +0000 UTC m=+876.055489848"
Oct 29 22:57:52.720010 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:52.719977    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:57:52.721590 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:52.721566    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:57:52.721713 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:52.721612    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm" podUID="e415acf5-67c8-47c1-aa03-df58dca41b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:57:52.806951 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:52.806917    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:57:52.808349 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:52.808319    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 29 22:57:52.808486 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:52.808376    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z" podUID="b20a2142-6441-4846-8855-b4595d2fe746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 29 22:57:53.240825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:53.240401    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:57:53.240825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:53.240465    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n" podUID="7c5d481f-83bd-48f3-8945-cd88d304e8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 29 22:57:53.720824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:53.720225    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:57:53.720824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:53.720293    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm" podUID="e415acf5-67c8-47c1-aa03-df58dca41b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:57:53.808826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:53.808693    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 29 22:57:53.808826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:53.808755    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z" podUID="b20a2142-6441-4846-8855-b4595d2fe746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 29 22:57:54.210585 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:54.210544    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:57:54.211041 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:54.210915    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n" podUID="7c5d481f-83bd-48f3-8945-cd88d304e8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 29 22:57:54.213899 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:54.213719    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:54.343846 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:54.343796    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:54.497938 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:54.497847    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:54.724829 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:54.722171    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:57:54.724829 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:54.722231    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm" podUID="e415acf5-67c8-47c1-aa03-df58dca41b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:57:54.822726 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:54.822633    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 29 22:57:54.823042 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:54.823013    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z" podUID="b20a2142-6441-4846-8855-b4595d2fe746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 29 22:57:55.202241 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:55.202204    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:57:55.202525 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:55.202492    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n" podUID="7c5d481f-83bd-48f3-8945-cd88d304e8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 29 22:57:55.721325 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:55.721282    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:57:55.721506 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:55.721358    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm" podUID="e415acf5-67c8-47c1-aa03-df58dca41b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:57:55.805506 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:55.805465    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 29 22:57:55.806140 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:55.806105    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z" podUID="b20a2142-6441-4846-8855-b4595d2fe746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 29 22:57:56.037160 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:56.037073    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:56.153693 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:56.153663    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:56.220069 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:56.220027    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:57:56.220374 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:56.220343    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n" podUID="7c5d481f-83bd-48f3-8945-cd88d304e8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 29 22:57:56.310509 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:56.301520    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:56.724826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:56.721411    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:57:56.724826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:56.721477    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm" podUID="e415acf5-67c8-47c1-aa03-df58dca41b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:57:56.824826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:56.816731    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 29 22:57:56.824826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:56.816818    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z" podUID="b20a2142-6441-4846-8855-b4595d2fe746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 29 22:57:57.333849 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:57.331644    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:57.333849 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:57.333255    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:57.335878 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:57.335147    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 22:57:57.720481 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:57.720446    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:57:57.720790 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:57.720750    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm" podUID="e415acf5-67c8-47c1-aa03-df58dca41b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:57:57.815829 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:57.806236    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 29 22:57:57.815829 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:57.806329    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z" podUID="b20a2142-6441-4846-8855-b4595d2fe746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 29 22:57:57.931390 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:57.931353    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:58.119648 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:58.119590    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:58.719535 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:58.719496    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:57:58.719840 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:58.719775    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm" podUID="e415acf5-67c8-47c1-aa03-df58dca41b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:57:58.805527 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:58.805495    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 29 22:57:58.805710 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:58.805551    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z" podUID="b20a2142-6441-4846-8855-b4595d2fe746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 29 22:57:59.719574 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:59.719532    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:57:59.720105 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:59.719614    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm" podUID="e415acf5-67c8-47c1-aa03-df58dca41b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 29 22:57:59.996680 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:57:59.996566    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:00.239541 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:00.239499    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:58:00.241528 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:00.241505    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:58:00.245179 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:00.245153    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 22:58:00.724337 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:00.724300    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:58:00.724819 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:00.724781    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:58:00.725457 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:00.725433    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 22:58:00.861987 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:00.861944    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:01.753289 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:01.753255    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:03.383645 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:03.383605    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:05.054725 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:05.054691    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:06.796721 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:06.796675    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:07.043185 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.043148    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"]
Oct 29 22:58:07.048997 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.048942    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.084957 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.084922    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5rck\" (UniqueName: \"kubernetes.io/projected/16750225-7382-4170-8297-4def491a1ede-kube-api-access-t5rck\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.085179 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.084975    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.085179 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.085072    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16750225-7382-4170-8297-4def491a1ede-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.085357 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.085194    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16750225-7382-4170-8297-4def491a1ede-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.085416 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.085376    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.085467 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.085428    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.085543 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.085522    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16750225-7382-4170-8297-4def491a1ede-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.085620 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.085602    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.085956 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.085923    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.100943 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.100913    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"]
Oct 29 22:58:07.187810 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.187775    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.188007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.187876    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rck\" (UniqueName: \"kubernetes.io/projected/16750225-7382-4170-8297-4def491a1ede-kube-api-access-t5rck\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.188007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.187901    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.188133 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.188023    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16750225-7382-4170-8297-4def491a1ede-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.188133 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.188111    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16750225-7382-4170-8297-4def491a1ede-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.188239 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.188201    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.188292 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.188262    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.188352 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.188270    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.188352 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.188318    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.188461 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.188383    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16750225-7382-4170-8297-4def491a1ede-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.188461 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.188437    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.188823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.188782    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.188999 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.188848    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.189271 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.189241    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16750225-7382-4170-8297-4def491a1ede-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.190677 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.190650    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.190818 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.190788    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16750225-7382-4170-8297-4def491a1ede-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.209360 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.209334    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5rck\" (UniqueName: \"kubernetes.io/projected/16750225-7382-4170-8297-4def491a1ede-kube-api-access-t5rck\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.209442 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.209352    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16750225-7382-4170-8297-4def491a1ede-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.365355 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.365324    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:07.697196 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.697165    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"]
Oct 29 22:58:07.698108 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:58:07.698072    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16750225_7382_4170_8297_4def491a1ede.slice/crio-4a7d9dffca273715b8794bb51e57cc7d9cbc8007764babecd58896277c67cc94 WatchSource:0}: Error finding container 4a7d9dffca273715b8794bb51e57cc7d9cbc8007764babecd58896277c67cc94: Status 404 returned error can't find the container with id 4a7d9dffca273715b8794bb51e57cc7d9cbc8007764babecd58896277c67cc94
Oct 29 22:58:07.701029 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.700991    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:07.701109 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.701061    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:07.701109 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.701096    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:07.807623 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.807586    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk" event={"ID":"16750225-7382-4170-8297-4def491a1ede","Type":"ContainerStarted","Data":"bbccb5b86a5de6419444b08228a2af060bcc498019b5069073ac8151670bcb34"}
Oct 29 22:58:07.808068 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.807656    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk" event={"ID":"16750225-7382-4170-8297-4def491a1ede","Type":"ContainerStarted","Data":"4a7d9dffca273715b8794bb51e57cc7d9cbc8007764babecd58896277c67cc94"}
Oct 29 22:58:07.853254 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:07.853200    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk" podStartSLOduration=0.853186834 podStartE2EDuration="853.186834ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:07.848497891 +0000 UTC m=+891.191446636" watchObservedRunningTime="2026-10-29 22:58:07.853186834 +0000 UTC m=+891.196135638"
Oct 29 22:58:08.366023 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:08.365991    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:08.367441 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:08.367414    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:58:08.367561 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:08.367475    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk" podUID="16750225-7382-4170-8297-4def491a1ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 29 22:58:08.584772 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:08.584724    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:09.379837 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:09.378167    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:58:09.379837 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:09.378229    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk" podUID="16750225-7382-4170-8297-4def491a1ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 29 22:58:10.047833 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.047769    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"]
Oct 29 22:58:10.052411 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.052391    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.097777 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.097746    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"]
Oct 29 22:58:10.143037 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.143003    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.143183 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.143061    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.143183 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.143143    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.143388 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.143366    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.143471 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.143429    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5mqq\" (UniqueName: \"kubernetes.io/projected/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-kube-api-access-p5mqq\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.143516 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.143500    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.143634 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.143614    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.143754 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.143737    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.143823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.143793    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.245705 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.245668    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.245910 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.245728    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5mqq\" (UniqueName: \"kubernetes.io/projected/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-kube-api-access-p5mqq\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.245910 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.245751    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.245910 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.245836    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.246117 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.246016    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.246117 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.246100    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.246217 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.246166    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.246286 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.246263    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.246286 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.246273    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.246403 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.246272    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.246403 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.246334    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.246403 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.246351    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.246641 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.246624    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.246687 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.246668    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.248652 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.248624    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.248652 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.248628    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.291061 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.291019    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.292703 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.292671    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5mqq\" (UniqueName: \"kubernetes.io/projected/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-kube-api-access-p5mqq\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.384440 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.383896    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:58:10.384440 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.383961    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk" podUID="16750225-7382-4170-8297-4def491a1ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 29 22:58:10.393662 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.393635    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:10.693376 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.693284    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:10.726692 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.726661    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"]
Oct 29 22:58:10.727343 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:58:10.727311    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0855c417_df52_4f3d_8a49_bc3b1fb88a0f.slice/crio-15969f6c3f6e35e9a70f44794fde01534c703cb83665834127b745eb3709c8dc WatchSource:0}: Error finding container 15969f6c3f6e35e9a70f44794fde01534c703cb83665834127b745eb3709c8dc: Status 404 returned error can't find the container with id 15969f6c3f6e35e9a70f44794fde01534c703cb83665834127b745eb3709c8dc
Oct 29 22:58:10.730215 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.730177    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:10.730308 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.730263    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:10.730366 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.730307    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:10.809097 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.809067    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:10.847823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:10.847781    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p" event={"ID":"0855c417-df52-4f3d-8a49-bc3b1fb88a0f","Type":"ContainerStarted","Data":"15969f6c3f6e35e9a70f44794fde01534c703cb83665834127b745eb3709c8dc"}
Oct 29 22:58:11.383868 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:11.371219    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:58:11.383868 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:11.371282    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk" podUID="16750225-7382-4170-8297-4def491a1ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 29 22:58:11.407547 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:11.407512    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:11.926849 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:11.925703    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p" event={"ID":"0855c417-df52-4f3d-8a49-bc3b1fb88a0f","Type":"ContainerStarted","Data":"dc0e9f6496bbd8ea4fd7921f803ad38eb4abd52b6bbab2c763eeaaeb55403585"}
Oct 29 22:58:12.025893 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:12.025822    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p" podStartSLOduration=2.025780706 podStartE2EDuration="2.025780706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:11.988653638 +0000 UTC m=+895.331602384" watchObservedRunningTime="2026-10-29 22:58:12.025780706 +0000 UTC m=+895.368729450"
Oct 29 22:58:12.377965 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:12.376684    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:58:12.377965 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:12.376759    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk" podUID="16750225-7382-4170-8297-4def491a1ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 29 22:58:12.399883 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:12.397371    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:12.409709 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:12.408216    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:58:12.409709 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:12.408283    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p" podUID="0855c417-df52-4f3d-8a49-bc3b1fb88a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 29 22:58:13.016924 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:13.016024    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:13.370845 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:13.367571    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:58:13.370845 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:13.367634    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk" podUID="16750225-7382-4170-8297-4def491a1ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 29 22:58:13.396828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:13.395606    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:58:13.396828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:13.395669    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p" podUID="0855c417-df52-4f3d-8a49-bc3b1fb88a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 29 22:58:13.665827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:13.656151    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"]
Oct 29 22:58:13.732826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:13.719477    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:13.767826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:13.746112    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-79e6c631\"/\"default-dockercfg-f292x\""
Oct 29 22:58:13.981147 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:13.973669    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-79e6c631\"/\"kube-root-ca.crt\""
Oct 29 22:58:14.020439 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:14.007661    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-79e6c631\"/\"openshift-service-ca.crt\""
Oct 29 22:58:14.020439 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:14.008612    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-79e6c631\"/\"e2e-lifecycle-model-79e6c631-kserve-self-signed-certs\""
Oct 29 22:58:14.263032 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:14.262897    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:14.282560 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:14.282520    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-dshm\") pod \"e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") " pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:14.290836 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:14.282898    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-tls-certs\") pod \"e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") " pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:14.393826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:14.390052    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:58:14.409833 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:14.401936    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk" podUID="16750225-7382-4170-8297-4def491a1ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 29 22:58:14.622186 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:14.582490    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"]
Oct 29 22:58:14.622186 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:14.582622    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:14.622186 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:14.583010    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:58:14.622186 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:14.589896    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:14.756920 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:14.756865    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-model-cache\") pod \"e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") " pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:14.757960 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:14.757933    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-tmp-dir\") pod \"e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") " pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:14.778827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:14.758182    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b6xp\" (UniqueName: \"kubernetes.io/projected/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-kube-api-access-9b6xp\") pod \"e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") " pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:14.781308 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:14.781258    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-home\") pod \"e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") " pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:14.831454 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:14.831417    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:58:14.831837 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:14.831784    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p" podUID="0855c417-df52-4f3d-8a49-bc3b1fb88a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 29 22:58:15.042887 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.038787    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:58:15.065827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.065023    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:15.143833 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.118505    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-model-cache\") pod \"e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") " pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:15.246823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.246443    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-tmp-dir\") pod \"e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") " pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:15.246823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.246525    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b6xp\" (UniqueName: \"kubernetes.io/projected/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-kube-api-access-9b6xp\") pod \"e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") " pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:15.246823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.246659    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-home\") pod \"e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") " pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:15.247131 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.246927    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-dshm\") pod \"e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") " pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:15.247131 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.246994    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-tls-certs\") pod \"e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") " pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:15.261862 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.145735    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-model-cache\") pod \"e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") " pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:15.261862 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.248036    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-tmp-dir\") pod \"e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") " pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:15.262756 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.262724    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-home\") pod \"e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") " pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:15.308915 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.308771    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b6xp\" (UniqueName: \"kubernetes.io/projected/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-kube-api-access-9b6xp\") pod \"e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") " pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:15.317021 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.309390    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-tls-certs\") pod \"e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") " pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:15.317021 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.315913    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:15.317021 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.316212    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-dshm\") pod \"e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") " pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:15.362456 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.362407    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:15.368323 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.368290    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:58:15.368549 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.368516    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk" podUID="16750225-7382-4170-8297-4def491a1ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 29 22:58:15.436964 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.436930    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:58:15.439534 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.439489    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p" podUID="0855c417-df52-4f3d-8a49-bc3b1fb88a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 29 22:58:15.595009 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.569975    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:15.595009 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.584906    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:15.595009 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.591330    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:15.595009 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.592268    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:58:15.595009 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.592336    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:58:15.623827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.595964    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:15.623827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.596372    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:58:15.826991 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:15.810373    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:16.117884 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:16.087390    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:58:16.117884 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:16.089594    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:58:16.117884 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:16.103078    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:16.223733 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:16.223687    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:16.330826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:16.305895    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:16.330826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:16.305958    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z" podUID="b20a2142-6441-4846-8855-b4595d2fe746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:16.357838 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:16.347233    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:16.357838 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:16.347606    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:58:16.448932 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:16.413262    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:58:16.448932 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:16.413323    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p" podUID="0855c417-df52-4f3d-8a49-bc3b1fb88a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 29 22:58:16.593825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:16.586334    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:16.614940 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:16.614889    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:58:16.617422 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:16.597995    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:58:16.676827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:16.651880    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:16.986878 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:16.831890    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 29 22:58:16.986878 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:16.835086    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:17.111827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:17.073360    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:17.111827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:17.073421    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm" podUID="e415acf5-67c8-47c1-aa03-df58dca41b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:17.141828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:17.137717    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:58:17.186829 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:17.169633    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:17.186829 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:17.173965    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x" podUID="75a16c52-71c1-4da1-bc73-4e6fea6184c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:17.188421 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:17.188385    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:58:17.480071 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:17.458512    5428 scope.go:122] "RemoveContainer" containerID="3d8f14cdf8569f4e26ba97511266a981b6549d5d9d5ecb1af858c716f5f18e1b"
Oct 29 22:58:17.510603 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:17.510565    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 29 22:58:17.667503 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:17.588360    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:58:17.845678 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:17.845641    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p" podUID="0855c417-df52-4f3d-8a49-bc3b1fb88a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 29 22:58:17.846897 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:17.659129    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:17.847242 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:17.659165    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:58:17.847539 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:17.838172    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:17.847816 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:17.657775    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:17.848112 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:17.656348    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:17.851883 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:17.851850    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk" podUID="16750225-7382-4170-8297-4def491a1ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:18.297847 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:18.296868    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:19.150404 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:19.150349    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:58:19.172426 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:19.160041    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 29 22:58:19.172426 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:19.166721    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:58:19.174550 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:18.568035    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:19.174940 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:19.174905    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm" podUID="9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:19.214851 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:18.601431    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 29 22:58:19.375007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:19.287679    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:19.375007 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:58:19.345245    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:58:19.375007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:19.352993    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:58:19.375007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:19.353089    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:58:19.375007 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:19.368839    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p" podUID="0855c417-df52-4f3d-8a49-bc3b1fb88a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 29 22:58:19.446539 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:19.446500    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:19.446836 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:19.446788    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk" podUID="16750225-7382-4170-8297-4def491a1ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:19.474421 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:19.471935    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:19.522857 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:19.508164    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc" podUID="c4034d76-e0f2-4674-928d-657820813e88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 29 22:58:19.896825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:19.896136    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:19.992824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:19.986681    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:58:19.992824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:19.986759    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p" podUID="0855c417-df52-4f3d-8a49-bc3b1fb88a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 29 22:58:20.220887 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.183683    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc" podUID="87e30ba6-6bca-4971-844c-c9509f5b0d0b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:20.256990 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.256711    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 22:58:20.261988 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.257557    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 29 22:58:20.261988 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.258326    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:58:20.425217 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.425181    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:58:20.425714 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.425338    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p" podUID="0855c417-df52-4f3d-8a49-bc3b1fb88a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 29 22:58:20.430048 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.430004    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 29 22:58:20.439747 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.439636    5428 scope.go:122] "RemoveContainer" containerID="476b2a838007749fe4875e03a56f07b0c56cfef438771702cea7359a7515dcdc"
Oct 29 22:58:20.453852 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:58:20.453826    5428 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.265s"
Oct 29 22:58:20.454498 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.454482    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:58:20.706849 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.689182    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:20.714401 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.714358    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj" event={"ID":"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db","Type":"ContainerStarted","Data":"cefa52c7e4fe27efa247e02a478faed6e66be591c5fb93dfcc81180fd692865d"}
Oct 29 22:58:20.714577 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.714475    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:20.714577 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.714534    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:58:20.714577 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.714570    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:58:20.714762 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.714608    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"]
Oct 29 22:58:20.717114 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.717079    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:20.723502 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.720516    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:58:20.734326 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.725956    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:58:20.734326 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.728554    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:58:20.734326 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.729560    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 22:58:20.837389 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:58:20.837332    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d8f14cdf8569f4e26ba97511266a981b6549d5d9d5ecb1af858c716f5f18e1b\": container with ID starting with 3d8f14cdf8569f4e26ba97511266a981b6549d5d9d5ecb1af858c716f5f18e1b not found: ID does not exist" containerID="3d8f14cdf8569f4e26ba97511266a981b6549d5d9d5ecb1af858c716f5f18e1b"
Oct 29 22:58:20.837581 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:58:20.837408    5428 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"3d8f14cdf8569f4e26ba97511266a981b6549d5d9d5ecb1af858c716f5f18e1b\": container with ID starting with 3d8f14cdf8569f4e26ba97511266a981b6549d5d9d5ecb1af858c716f5f18e1b not found: ID does not exist"
Oct 29 22:58:20.843189 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:20.843154    5428 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"3d8f14cdf8569f4e26ba97511266a981b6549d5d9d5ecb1af858c716f5f18e1b\": container with ID starting with 3d8f14cdf8569f4e26ba97511266a981b6549d5d9d5ecb1af858c716f5f18e1b not found: ID does not exist"
Oct 29 22:58:21.397733 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:21.394207    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:58:21.397733 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:21.394260    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p" podUID="0855c417-df52-4f3d-8a49-bc3b1fb88a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 29 22:58:21.738029 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:21.737941    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj" event={"ID":"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db","Type":"ContainerStarted","Data":"0a02f567e0ab7225097b4e06602220c64b6b3d71019d650a5ad524421dc6caa8"}
Oct 29 22:58:21.790882 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:21.790795    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj" podStartSLOduration=8.790775292 podStartE2EDuration="8.790775292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:21.778626237 +0000 UTC m=+905.121574983" watchObservedRunningTime="2026-10-29 22:58:21.790775292 +0000 UTC m=+905.133724037"
Oct 29 22:58:22.480827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:22.475127    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:22.480827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:22.476832    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:22.490281 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:22.490229    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 22:58:22.596117 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:22.596080    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:23.195303 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:23.195249    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:24.474036 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:24.473998    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:24.924215 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:24.924184    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:25.525771 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:25.525727    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:25.592596 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:25.592556    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:25.605507 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:25.605479    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:25.784255 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:25.784167    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:58:26.577668 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:26.577628    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:26.691650 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:26.691612    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:28.206900 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:28.206847    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:28.497151 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:28.497119    5428 patch_prober.go:29] interesting pod/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 29 22:58:28.497324 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:28.497170    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll" podUID="b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 29 22:58:28.800290 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:28.800204    5428 generic.go:356] "Generic (PLEG): container finished" podID="b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" containerID="f19ed61957bd77efa472c6ef591da14dac3b9e8abe6fb8c4e652edcc866c7d4e" exitCode=0
Oct 29 22:58:28.800290 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:28.800242    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll" event={"ID":"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff","Type":"ContainerDied","Data":"f19ed61957bd77efa472c6ef591da14dac3b9e8abe6fb8c4e652edcc866c7d4e"}
Oct 29 22:58:28.800684 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:28.800668    5428 scope.go:122] "RemoveContainer" containerID="f19ed61957bd77efa472c6ef591da14dac3b9e8abe6fb8c4e652edcc866c7d4e"
Oct 29 22:58:28.802234 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:28.802202    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:28.802322 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:28.802272    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:28.802322 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:28.802306    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:29.811965 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:29.811926    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll" event={"ID":"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff","Type":"ContainerStarted","Data":"e45987552fa8976f9c466a7e8f0717443c9563e07e22dae211aced093b1260f3"}
Oct 29 22:58:29.948890 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:29.948858    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:30.483977 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:30.483938    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:58:30.484239 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:30.484216    5428 patch_prober.go:29] interesting pod/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 29 22:58:30.484334 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:30.484283    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll" podUID="b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 29 22:58:31.484219 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:31.484180    5428 patch_prober.go:29] interesting pod/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 29 22:58:31.484724 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:31.484354    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll" podUID="b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 29 22:58:31.542349 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:31.542309    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:32.491080 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:32.491044    5428 patch_prober.go:29] interesting pod/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 29 22:58:32.491725 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:32.491694    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll" podUID="b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 29 22:58:33.489356 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:33.489318    5428 patch_prober.go:29] interesting pod/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 29 22:58:33.489640 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:33.489612    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll" podUID="b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 29 22:58:33.557877 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:33.557832    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:34.484318 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:34.484277    5428 patch_prober.go:29] interesting pod/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 29 22:58:34.484527 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:34.484334    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll" podUID="b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 29 22:58:35.094263 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.094213    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"]
Oct 29 22:58:35.101291 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.101255    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.105294 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.105261    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"]
Oct 29 22:58:35.145995 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.145960    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.146346 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.146321    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.146467 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.146368    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.146467 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.146393    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.146467 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.146418    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.146631 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.146472    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.146631 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.146522    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.146631 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.146546    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.146631 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.146607    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4p7s\" (UniqueName: \"kubernetes.io/projected/86892f1f-cbce-48a4-80ae-48e4c0f88abf-kube-api-access-c4p7s\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.252241 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.252199    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.253084 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.253030    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.253262 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.253243    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.253378 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.253363    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.253536 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.253520    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.254016 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.253616    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.254016 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.253318    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.254381 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.254359    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.254467 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.254418    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.254467 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.254446    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.254628 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.254608    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4p7s\" (UniqueName: \"kubernetes.io/projected/86892f1f-cbce-48a4-80ae-48e4c0f88abf-kube-api-access-c4p7s\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.254788 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.254773    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.255505 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.255481    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.256119 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.256097    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.288828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.284095    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.304433 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.304405    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.309956 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.309889    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.311964 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.311926    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4p7s\" (UniqueName: \"kubernetes.io/projected/86892f1f-cbce-48a4-80ae-48e4c0f88abf-kube-api-access-c4p7s\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.425735 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.425703    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:35.779507 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:58:35.779473    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86892f1f_cbce_48a4_80ae_48e4c0f88abf.slice/crio-0fed2c92b39a651931ba62a2283feb3d1dfa99dd0016b9b3cc6b19edaba7b5e7 WatchSource:0}: Error finding container 0fed2c92b39a651931ba62a2283feb3d1dfa99dd0016b9b3cc6b19edaba7b5e7: Status 404 returned error can't find the container with id 0fed2c92b39a651931ba62a2283feb3d1dfa99dd0016b9b3cc6b19edaba7b5e7
Oct 29 22:58:35.780272 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.780245    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"]
Oct 29 22:58:35.783624 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.783578    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:35.783730 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.783683    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:35.783780 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.783758    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:35.791170 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.791139    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:35.971857 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.964606    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx" event={"ID":"86892f1f-cbce-48a4-80ae-48e4c0f88abf","Type":"ContainerStarted","Data":"f1fd7a3a2fab6e67ffda72157861716c9386953c97576bc67b32628f2b0a7ff5"}
Oct 29 22:58:35.971857 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:35.964696    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx" event={"ID":"86892f1f-cbce-48a4-80ae-48e4c0f88abf","Type":"ContainerStarted","Data":"0fed2c92b39a651931ba62a2283feb3d1dfa99dd0016b9b3cc6b19edaba7b5e7"}
Oct 29 22:58:36.024825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:36.024337    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx" podStartSLOduration=1.024318508 podStartE2EDuration="1.024318508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:36.010206737 +0000 UTC m=+919.353155509" watchObservedRunningTime="2026-10-29 22:58:36.024318508 +0000 UTC m=+919.367267267"
Oct 29 22:58:36.097291 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:36.097207    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:58:36.098019 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:36.097995    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:58:36.099277 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:36.099255    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 22:58:36.428825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:36.426663    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:36.428825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:36.428428    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:58:36.428825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:36.428482    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx" podUID="86892f1f-cbce-48a4-80ae-48e4c0f88abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:58:37.427149 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:37.427116    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:58:37.427538 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:37.427187    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx" podUID="86892f1f-cbce-48a4-80ae-48e4c0f88abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:58:37.930779 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:37.930745    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:38.427132 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.427100    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:58:38.427302 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.427154    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx" podUID="86892f1f-cbce-48a4-80ae-48e4c0f88abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:58:38.658673 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.658642    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"]
Oct 29 22:58:38.684374 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.684228    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.694820 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.691382    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"]
Oct 29 22:58:38.697845 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.697662    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.702143 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.700008    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"]
Oct 29 22:58:38.730515 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.730479    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"]
Oct 29 22:58:38.731680 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.731534    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.731793 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.731745    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.731857 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.731827    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.731904 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.731891    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.731941 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.731924    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.731981 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.731951    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.732026 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.732009    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/101ae9b3-7447-4a91-b895-913c9a96f288-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.732067 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.732047    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dqf6\" (UniqueName: \"kubernetes.io/projected/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-kube-api-access-4dqf6\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.732128 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.732112    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.732179 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.732165    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101ae9b3-7447-4a91-b895-913c9a96f288-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.732238 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.732223    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101ae9b3-7447-4a91-b895-913c9a96f288-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.732286 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.732272    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.732354 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.732340    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.732406 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.732392    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmrdp\" (UniqueName: \"kubernetes.io/projected/101ae9b3-7447-4a91-b895-913c9a96f288-kube-api-access-nmrdp\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.732485 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.732469    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.732530 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.732515    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.732611 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.732550    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.732611 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.732585    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.833662 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.833625    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.833662 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.833673    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.833931 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.833701    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.833931 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.833753    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/101ae9b3-7447-4a91-b895-913c9a96f288-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.833931 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.833784    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dqf6\" (UniqueName: \"kubernetes.io/projected/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-kube-api-access-4dqf6\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.833931 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.833883    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.834151 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.833934    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101ae9b3-7447-4a91-b895-913c9a96f288-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.834151 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.833988    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101ae9b3-7447-4a91-b895-913c9a96f288-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.834151 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.834070    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.834151 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.834075    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.834151 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.834138    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.834421 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.834141    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.834421 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.834189    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.834421 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.834242    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmrdp\" (UniqueName: \"kubernetes.io/projected/101ae9b3-7447-4a91-b895-913c9a96f288-kube-api-access-nmrdp\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.834421 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.834316    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.834421 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.834388    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.834421 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.834419    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.834731 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.834452    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.834731 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.834494    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.834731 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.834504    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.834731 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.834615    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.834731 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.834668    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.834731 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.834699    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.835071 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.834855    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.835071 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.834871    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101ae9b3-7447-4a91-b895-913c9a96f288-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.835071 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.835010    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.835243 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.835101    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.835607 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.835584    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.836906 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.836877    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.837840 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.837133    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.837840 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.837454    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101ae9b3-7447-4a91-b895-913c9a96f288-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.837840 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.837470    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.855667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.855549    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/101ae9b3-7447-4a91-b895-913c9a96f288-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.856423 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.856397    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:38.856576 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.856555    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmrdp\" (UniqueName: \"kubernetes.io/projected/101ae9b3-7447-4a91-b895-913c9a96f288-kube-api-access-nmrdp\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:38.856621 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:38.856574    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dqf6\" (UniqueName: \"kubernetes.io/projected/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-kube-api-access-4dqf6\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:39.018711 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:39.018616    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:39.032216 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:39.032185    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:39.393915 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:39.393795    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"]
Oct 29 22:58:39.401086 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:58:39.401001    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101ae9b3_7447_4a91_b895_913c9a96f288.slice/crio-05b3478fd223a38743df7d4ae1e5d179988d2b59b0c6aa088d4e658956a76a6e WatchSource:0}: Error finding container 05b3478fd223a38743df7d4ae1e5d179988d2b59b0c6aa088d4e658956a76a6e: Status 404 returned error can't find the container with id 05b3478fd223a38743df7d4ae1e5d179988d2b59b0c6aa088d4e658956a76a6e
Oct 29 22:58:39.402461 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:58:39.402428    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c00e005_e8e5_4e90_9b0c_95d5d5de4952.slice/crio-d46e74fbba462cb930ef6e2fc21b4558db0fb581a8aa34d62be0aa87e13103b9 WatchSource:0}: Error finding container d46e74fbba462cb930ef6e2fc21b4558db0fb581a8aa34d62be0aa87e13103b9: Status 404 returned error can't find the container with id d46e74fbba462cb930ef6e2fc21b4558db0fb581a8aa34d62be0aa87e13103b9
Oct 29 22:58:39.402924 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:39.402902    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"]
Oct 29 22:58:39.405038 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:39.404955    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:39.405038 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:39.405026    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:39.405116 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:39.405060    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:39.405948 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:39.405917    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:39.406034 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:39.405979    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:39.406034 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:39.406008    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:39.426918 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:39.426893    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:58:39.427004 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:39.426941    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx" podUID="86892f1f-cbce-48a4-80ae-48e4c0f88abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:58:39.907714 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:39.907671    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:40.028508 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:40.027840    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn" event={"ID":"101ae9b3-7447-4a91-b895-913c9a96f288","Type":"ContainerStarted","Data":"a65733053bfe76faf24b4111e3e28b5c95ffb551cb39923b8ded2134bf8358a8"}
Oct 29 22:58:40.028508 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:40.027924    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn" event={"ID":"101ae9b3-7447-4a91-b895-913c9a96f288","Type":"ContainerStarted","Data":"05b3478fd223a38743df7d4ae1e5d179988d2b59b0c6aa088d4e658956a76a6e"}
Oct 29 22:58:40.033146 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:40.032663    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n" event={"ID":"4c00e005-e8e5-4e90-9b0c-95d5d5de4952","Type":"ContainerStarted","Data":"7cad7c838a179abf312b3e0a69cfd4753e40ee732ffbaf6e8cf150eaa8535c03"}
Oct 29 22:58:40.033146 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:40.032907    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n" event={"ID":"4c00e005-e8e5-4e90-9b0c-95d5d5de4952","Type":"ContainerStarted","Data":"d46e74fbba462cb930ef6e2fc21b4558db0fb581a8aa34d62be0aa87e13103b9"}
Oct 29 22:58:40.101460 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:40.101422    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:40.143837 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:40.141594    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn" podStartSLOduration=2.141574584 podStartE2EDuration="2.141574584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:40.091124932 +0000 UTC m=+923.434073678" watchObservedRunningTime="2026-10-29 22:58:40.141574584 +0000 UTC m=+923.484523329"
Oct 29 22:58:40.155823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:40.152013    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n" podStartSLOduration=2.151991536 podStartE2EDuration="2.151991536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:40.146403055 +0000 UTC m=+923.489351836" watchObservedRunningTime="2026-10-29 22:58:40.151991536 +0000 UTC m=+923.494940302"
Oct 29 22:58:40.427826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:40.427255    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:58:40.427826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:40.427319    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx" podUID="86892f1f-cbce-48a4-80ae-48e4c0f88abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:58:40.757697 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:40.757613    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:41.024882 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:41.024377    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:41.024882 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:41.024523    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:58:41.024882 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:41.024569    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn" podUID="101ae9b3-7447-4a91-b895-913c9a96f288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 29 22:58:41.036854 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:41.035906    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:41.036854 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:41.036479    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:58:41.036854 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:41.036532    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n" podUID="4c00e005-e8e5-4e90-9b0c-95d5d5de4952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 29 22:58:41.258719 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:41.256559    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"]
Oct 29 22:58:41.427827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:41.404606    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:41.587763 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:41.462487    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:58:41.587763 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:41.462555    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx" podUID="86892f1f-cbce-48a4-80ae-48e4c0f88abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:58:41.951636 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:41.951603    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"]
Oct 29 22:58:41.952360 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:41.952337    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.025826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.021318    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8b57a2-openshift-default-dockercfg-lj4wb\""
Oct 29 22:58:42.025826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.024008    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:58:42.025826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.024056    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn" podUID="101ae9b3-7447-4a91-b895-913c9a96f288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 29 22:58:42.044183 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.043517    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:58:42.044183 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.043571    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n" podUID="4c00e005-e8e5-4e90-9b0c-95d5d5de4952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 29 22:58:42.083826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.063917    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-istio-data\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.087951 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.087918    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1804496-00d1-4679-a069-89a1caec4e75-istio-token\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.088400 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.088380    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-istio-envoy\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.088617 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.088597    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxv47\" (UniqueName: \"kubernetes.io/projected/c1804496-00d1-4679-a069-89a1caec4e75-kube-api-access-mxv47\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.092943 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.092919    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-workload-socket\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.093315 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.093294    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-credential-socket\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.093609 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.093589    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1804496-00d1-4679-a069-89a1caec4e75-istio-podinfo\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.093777 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.093759    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1804496-00d1-4679-a069-89a1caec4e75-istiod-ca-cert\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.093977 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.093958    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-workload-certs\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.218133 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.218020    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-istio-data\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.219093 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.219067    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1804496-00d1-4679-a069-89a1caec4e75-istio-token\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.219255 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.219240    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-istio-envoy\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.219431 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.219411    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxv47\" (UniqueName: \"kubernetes.io/projected/c1804496-00d1-4679-a069-89a1caec4e75-kube-api-access-mxv47\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.219714 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.219689    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-workload-socket\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.220095 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.220068    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-credential-socket\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.220423 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.220402    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1804496-00d1-4679-a069-89a1caec4e75-istio-podinfo\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.220587 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.220570    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1804496-00d1-4679-a069-89a1caec4e75-istiod-ca-cert\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.220713 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.220698    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-workload-certs\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.221225 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.221198    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-workload-socket\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.221307 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.218983    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-istio-data\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.221453 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.221436    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-workload-certs\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.222436 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.222407    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-credential-socket\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.223406 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.223378    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1804496-00d1-4679-a069-89a1caec4e75-istiod-ca-cert\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.224590 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.224567    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-istio-envoy\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.224812 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.224776    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1804496-00d1-4679-a069-89a1caec4e75-istio-podinfo\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.245150 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.245103    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1804496-00d1-4679-a069-89a1caec4e75-istio-token\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.245762 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.245737    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxv47\" (UniqueName: \"kubernetes.io/projected/c1804496-00d1-4679-a069-89a1caec4e75-kube-api-access-mxv47\") pod \"e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") " pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.246228 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.246197    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:42.351052 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.351018    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:42.403031 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.402225    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:42.427297 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.427178    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:58:42.427297 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.427287    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx" podUID="86892f1f-cbce-48a4-80ae-48e4c0f88abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:58:42.984838 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.975320    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"]
Oct 29 22:58:43.016015 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:42.999966    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:43.016015 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:43.000058    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:43.016015 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:43.000102    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:58:43.020210 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:43.019920    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:58:43.020210 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:43.019974    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn" podUID="101ae9b3-7447-4a91-b895-913c9a96f288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 29 22:58:43.033149 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:43.033123    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:58:43.033454 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:43.033176    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n" podUID="4c00e005-e8e5-4e90-9b0c-95d5d5de4952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 29 22:58:43.127835 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:43.123125    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p" event={"ID":"c1804496-00d1-4679-a069-89a1caec4e75","Type":"ContainerStarted","Data":"b7a137c30a0b66c8bb496a69b1937d5b3b0ddf97ce546e4b3bdbc20535dfc6d0"}
Oct 29 22:58:43.427178 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:43.427134    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:58:43.427383 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:43.427195    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx" podUID="86892f1f-cbce-48a4-80ae-48e4c0f88abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 29 22:58:43.980826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:43.971262    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:44.038827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:44.027213    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:58:44.038827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:44.027279    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn" podUID="101ae9b3-7447-4a91-b895-913c9a96f288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 29 22:58:44.045871 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:44.040225    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:58:44.045871 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:44.040285    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n" podUID="4c00e005-e8e5-4e90-9b0c-95d5d5de4952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 29 22:58:44.247654 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:44.247562    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p" event={"ID":"c1804496-00d1-4679-a069-89a1caec4e75","Type":"ContainerStarted","Data":"24d1bd4a76f86cd41d477586472dd4f80b80aced81c3182a213ec406bf54a425"}
Oct 29 22:58:44.331204 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:44.331135    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p" podStartSLOduration=3.331113123 podStartE2EDuration="3.331113123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:44.309341194 +0000 UTC m=+927.652289939" watchObservedRunningTime="2026-10-29 22:58:44.331113123 +0000 UTC m=+927.674061884"
Oct 29 22:58:44.352214 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:44.352174    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:44.357587 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:44.357561    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:44.431575 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:44.431544    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:44.433543 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:44.432348    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:44.437816 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:44.435286    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 22:58:45.019381 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:45.019341    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:58:45.019592 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:45.019402    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn" podUID="101ae9b3-7447-4a91-b895-913c9a96f288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 29 22:58:45.033415 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:45.033382    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:58:45.033561 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:45.033441    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n" podUID="4c00e005-e8e5-4e90-9b0c-95d5d5de4952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 29 22:58:45.255474 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:45.255442    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:45.256513 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:45.256493    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:58:46.019994 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:46.019946    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 29 22:58:46.020179 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:46.020012    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn" podUID="101ae9b3-7447-4a91-b895-913c9a96f288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 29 22:58:46.033461 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:46.033426    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 29 22:58:46.033630 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:46.033484    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n" podUID="4c00e005-e8e5-4e90-9b0c-95d5d5de4952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 29 22:58:46.043571 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:46.043532    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:46.737734 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:46.737698    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:46.855205 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:46.855163    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:47.391782 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:58:47.391713    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:58:47.488491 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:47.488458    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:47.489344 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:47.489321    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:47.490236 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:47.490212    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 22:58:47.651556 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:47.650183    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:47.652472 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:47.652153    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:47.668887 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:47.652981    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 22:58:48.060043 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:48.059957    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:48.449904 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:48.449859    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:48.589920 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:48.589874    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:49.991045 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:49.991010    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:51.753427 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:51.753393    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:53.599103 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:53.599072    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:55.498458 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:55.498418    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:57.175038 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:57.175003    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:59.097757 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:58:59.097715    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:00.879357 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:00.879317    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:02.661082 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:02.661044    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:03.269287 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:03.269254    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:03.378751 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:03.378718    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:04.292558 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:04.292522    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:04.417369 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:04.417322    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:04.538546 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:04.538508    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:04.688942 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:04.688899    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:06.402731 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:06.402696    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:08.044427 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:08.044398    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:09.703026 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:09.702986    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:10.328251 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:10.328218    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:11.355462 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:11.355428    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:11.654588 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:11.654558    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:12.074368 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.074279    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"]
Oct 29 22:59:12.082536 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.082506    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.108663 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.108627    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"]
Oct 29 22:59:12.110639 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.110613    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-479c2a-openshift-default-dockercfg-t2fmz\""
Oct 29 22:59:12.134669 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.134629    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:12.235111 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.226722    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-podinfo\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.235111 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.226916    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-workload-certs\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.235111 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.227230    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-workload-socket\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.235111 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.227426    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-token\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.235111 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.227468    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-credential-socket\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.235111 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.228246    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25gql\" (UniqueName: \"kubernetes.io/projected/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-kube-api-access-25gql\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.235111 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.228419    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-data\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.235111 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.228551    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-envoy\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.235111 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.228635    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istiod-ca-cert\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.315823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.288630    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:12.330102 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.329997    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-podinfo\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.330382 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.330358    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-workload-certs\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.330569 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.330552    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-workload-socket\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.330720 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.330701    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-token\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.342700 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.342670    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-credential-socket\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.342981 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.342959    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25gql\" (UniqueName: \"kubernetes.io/projected/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-kube-api-access-25gql\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.343253 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.343231    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-data\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.343482 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.343461    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-envoy\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.343661 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.343645    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istiod-ca-cert\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.344941 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.344569    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istiod-ca-cert\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.353937 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.332861    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-workload-certs\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.354155 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.345565    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-credential-socket\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.354275 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.345749    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-data\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.355068 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.355035    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-workload-socket\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.380536 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.380501    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-envoy\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.402833 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.400580    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-podinfo\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.418404 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.418370    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-token\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.419165 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.419137    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25gql\" (UniqueName: \"kubernetes.io/projected/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-kube-api-access-25gql\") pod \"e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") " pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:12.706990 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:12.706956    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:13.105142 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:13.105107    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"]
Oct 29 22:59:13.105481 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:59:13.105450    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0fa5ed2_1b9e_41e7_8d0d_e56a7c2f1763.slice/crio-1058ce3cda78c3714930ee5801f935e0d2af6dd5e3e5f5c6c6c0e2bd182b1236 WatchSource:0}: Error finding container 1058ce3cda78c3714930ee5801f935e0d2af6dd5e3e5f5c6c6c0e2bd182b1236: Status 404 returned error can't find the container with id 1058ce3cda78c3714930ee5801f935e0d2af6dd5e3e5f5c6c6c0e2bd182b1236
Oct 29 22:59:13.108518 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:13.108481    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:59:13.108585 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:13.108563    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:59:13.108625 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:13.108608    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:59:13.243013 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:13.242975    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:13.585671 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:13.585623    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9" event={"ID":"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763","Type":"ContainerStarted","Data":"27660b71d8a1de8e19b5f7b9b3431863b594c9aaa8e92e8fcf2aaef13c9bf66b"}
Oct 29 22:59:13.586184 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:13.585697    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9" event={"ID":"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763","Type":"ContainerStarted","Data":"1058ce3cda78c3714930ee5801f935e0d2af6dd5e3e5f5c6c6c0e2bd182b1236"}
Oct 29 22:59:13.633515 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:13.633459    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9" podStartSLOduration=1.633444101 podStartE2EDuration="1.633444101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:13.626793345 +0000 UTC m=+956.969742091" watchObservedRunningTime="2026-10-29 22:59:13.633444101 +0000 UTC m=+956.976392846"
Oct 29 22:59:13.707863 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:13.707833    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:14.713917 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:14.713889    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:14.911782 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:14.911747    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:15.600550 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:15.600516    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:15.601753 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:15.601724    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 22:59:15.988615 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:15.988578    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:16.107302 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:16.107252    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:16.984636 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:16.984606    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:17.368601 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:59:17.368568    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:59:17.752516 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:17.752368    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:17.874963 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:17.874925    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:18.976426 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:18.976364    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:20.681940 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:20.681896    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:21.280787 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.280747    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:21.695503 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.695469    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"]
Oct 29 22:59:21.702728 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.702700    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.717443 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.717419    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-660540-openshift-default-dockercfg-qw84q\""
Oct 29 22:59:21.735144 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.735115    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"]
Oct 29 22:59:21.817050 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.816261    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.817050 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.816317    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.817050 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.816534    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.817050 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.816571    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.817050 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.816640    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.817050 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.816665    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.817050 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.816737    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vfwv\" (UniqueName: \"kubernetes.io/projected/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-kube-api-access-9vfwv\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.817050 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.816993    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.817652 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.817160    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.927824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.926126    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.949869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.926194    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.949869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.941215    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.949869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.941254    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.949869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.941347    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vfwv\" (UniqueName: \"kubernetes.io/projected/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-kube-api-access-9vfwv\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.949869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.941460    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.949869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.941519    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.949869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.941670    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.949869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.941707    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.949869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.943636    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.949869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.943956    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.949869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.944350    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.949869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.944597    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.949869 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.945263    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.981460 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.981423    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:21.997446 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.995632    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:21.998673 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:21.998644    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vfwv\" (UniqueName: \"kubernetes.io/projected/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-kube-api-access-9vfwv\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:22.003855 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:22.000056    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:22.003855 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:22.001043    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-h2752\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:22.022146 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:22.021845    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:22.103426 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:22.103391    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:22.380994 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:59:22.380953    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eeb8c61_ed7d_420a_b256_e14c3a750d9b.slice/crio-e4d4d962ec02c9dd9f6bac27b99d7d70f1be9bbe5d762e8ea2cb4df48daaca76 WatchSource:0}: Error finding container e4d4d962ec02c9dd9f6bac27b99d7d70f1be9bbe5d762e8ea2cb4df48daaca76: Status 404 returned error can't find the container with id e4d4d962ec02c9dd9f6bac27b99d7d70f1be9bbe5d762e8ea2cb4df48daaca76
Oct 29 22:59:22.384305 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:22.384272    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"]
Oct 29 22:59:22.384498 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:22.384289    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:59:22.384653 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:22.384499    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:59:22.384653 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:22.384555    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:59:22.655246 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:22.655136    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752" event={"ID":"9eeb8c61-ed7d-420a-b256-e14c3a750d9b","Type":"ContainerStarted","Data":"6608e117ee2250f826cadcc1a42e0eb48748a583a001124ba960d7039eabf579"}
Oct 29 22:59:22.655408 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:22.655244    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752" event={"ID":"9eeb8c61-ed7d-420a-b256-e14c3a750d9b","Type":"ContainerStarted","Data":"e4d4d962ec02c9dd9f6bac27b99d7d70f1be9bbe5d762e8ea2cb4df48daaca76"}
Oct 29 22:59:22.700840 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:22.700774    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752" podStartSLOduration=1.700759121 podStartE2EDuration="1.700759121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:22.696066906 +0000 UTC m=+966.039015651" watchObservedRunningTime="2026-10-29 22:59:22.700759121 +0000 UTC m=+966.043707866"
Oct 29 22:59:22.803099 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:22.803066    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:23.023204 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:23.023113    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:23.029050 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:23.029025    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:23.663343 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:23.663314    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:23.664230 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:23.664210    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 22:59:24.574551 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:24.574517    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:24.776261 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:24.776222    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:25.178690 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:25.178648    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:25.288124 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:25.288092    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:25.640187 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:25.640154    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:26.627398 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:26.627362    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:27.389724 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:27.389685    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:27.810777 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:27.810685    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:27.925151 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:27.925116    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:27.989592 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:27.989560    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr"]
Oct 29 22:59:28.002638 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:28.002611    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr"
Oct 29 22:59:28.019237 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:28.019203    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr"]
Oct 29 22:59:28.059230 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:28.059190    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-660540-dockercfg-29vc2\""
Oct 29 22:59:28.073881 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:28.070532    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-660540\""
Oct 29 22:59:28.158084 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:28.142344    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c47ec09e-792a-4344-af69-0d85b3f64125-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr\" (UID: \"c47ec09e-792a-4344-af69-0d85b3f64125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr"
Oct 29 22:59:28.158084 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:28.142702    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dv4d\" (UniqueName: \"kubernetes.io/projected/c47ec09e-792a-4344-af69-0d85b3f64125-kube-api-access-4dv4d\") pod \"payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr\" (UID: \"c47ec09e-792a-4344-af69-0d85b3f64125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr"
Oct 29 22:59:28.279845 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:28.276284    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c47ec09e-792a-4344-af69-0d85b3f64125-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr\" (UID: \"c47ec09e-792a-4344-af69-0d85b3f64125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr"
Oct 29 22:59:28.279845 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:28.276504    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dv4d\" (UniqueName: \"kubernetes.io/projected/c47ec09e-792a-4344-af69-0d85b3f64125-kube-api-access-4dv4d\") pod \"payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr\" (UID: \"c47ec09e-792a-4344-af69-0d85b3f64125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr"
Oct 29 22:59:28.279845 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:28.277777    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c47ec09e-792a-4344-af69-0d85b3f64125-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr\" (UID: \"c47ec09e-792a-4344-af69-0d85b3f64125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr"
Oct 29 22:59:28.312821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:28.312525    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dv4d\" (UniqueName: \"kubernetes.io/projected/c47ec09e-792a-4344-af69-0d85b3f64125-kube-api-access-4dv4d\") pod \"payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr\" (UID: \"c47ec09e-792a-4344-af69-0d85b3f64125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr"
Oct 29 22:59:28.329846 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:28.329718    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr"
Oct 29 22:59:28.691377 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:28.691340    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr"]
Oct 29 22:59:28.692994 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:59:28.692961    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47ec09e_792a_4344_af69_0d85b3f64125.slice/crio-5ce57b8fd634c2be94147d4cf7bd1a5723ad65c43a24d66ec0338d04ae039b5c WatchSource:0}: Error finding container 5ce57b8fd634c2be94147d4cf7bd1a5723ad65c43a24d66ec0338d04ae039b5c: Status 404 returned error can't find the container with id 5ce57b8fd634c2be94147d4cf7bd1a5723ad65c43a24d66ec0338d04ae039b5c
Oct 29 22:59:28.712209 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:28.712176    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr" event={"ID":"c47ec09e-792a-4344-af69-0d85b3f64125","Type":"ContainerStarted","Data":"5ce57b8fd634c2be94147d4cf7bd1a5723ad65c43a24d66ec0338d04ae039b5c"}
Oct 29 22:59:29.036334 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:29.036249    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:29.722745 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:29.722708    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr" event={"ID":"c47ec09e-792a-4344-af69-0d85b3f64125","Type":"ContainerStarted","Data":"c80493c215328b75783bd1610ec632248002fc3f04d825f75db0ee8ff6cd0e76"}
Oct 29 22:59:29.723266 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:29.722796    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr"
Oct 29 22:59:29.761964 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:29.761907    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr" podStartSLOduration=2.76189207 podStartE2EDuration="2.76189207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:29.759394199 +0000 UTC m=+973.102342945" watchObservedRunningTime="2026-10-29 22:59:29.76189207 +0000 UTC m=+973.104840814"
Oct 29 22:59:30.864580 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:30.864535    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:30.945344 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:30.945305    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"]
Oct 29 22:59:30.949975 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:30.949954    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:30.993882 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:30.988358    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"]
Oct 29 22:59:31.033248 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.033210    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.033248 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.033254    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.033577 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.033286    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.033577 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.033304    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.033577 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.033358    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.033577 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.033418    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.033577 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.033488    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.034251 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.033593    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.034251 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.033659    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkqbk\" (UniqueName: \"kubernetes.io/projected/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-kube-api-access-mkqbk\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.137880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.136930    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkqbk\" (UniqueName: \"kubernetes.io/projected/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-kube-api-access-mkqbk\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.137880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.137083    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.137880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.137110    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.137880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.137154    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.137880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.137181    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.137880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.137255    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.137880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.137342    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.137880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.137451    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.137880 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.137590    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.138476 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.138421    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.145823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.138696    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.145823 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.138988    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.161866 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.152584    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.161866 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.154174    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.169820 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.166094    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.181785 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.174982    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.214821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.214668    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkqbk\" (UniqueName: \"kubernetes.io/projected/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-kube-api-access-mkqbk\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.241826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.239876    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.276277 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.275013    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:31.704239 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.704211    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"]
Oct 29 22:59:31.704990 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:59:31.704966    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ce5e15_2b45_4022_9945_04f4a9d3dd5d.slice/crio-532059e62c9e83d4a103c94e8927273c0df93654f0873a4105ffa3532b183646 WatchSource:0}: Error finding container 532059e62c9e83d4a103c94e8927273c0df93654f0873a4105ffa3532b183646: Status 404 returned error can't find the container with id 532059e62c9e83d4a103c94e8927273c0df93654f0873a4105ffa3532b183646
Oct 29 22:59:31.708273 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.708237    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:59:31.708348 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.708329    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:59:31.708388 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.708372    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:59:31.740470 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:31.740431    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4" event={"ID":"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d","Type":"ContainerStarted","Data":"532059e62c9e83d4a103c94e8927273c0df93654f0873a4105ffa3532b183646"}
Oct 29 22:59:32.749412 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:32.749372    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4" event={"ID":"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d","Type":"ContainerStarted","Data":"6a1eb9df47be2291edee463dbb75f8b1df9999d4e1c79f0493a4dc4d3c22c7ce"}
Oct 29 22:59:32.823579 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:32.823518    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4" podStartSLOduration=2.8234988210000003 podStartE2EDuration="2.823498821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:32.810903529 +0000 UTC m=+976.153852276" watchObservedRunningTime="2026-10-29 22:59:32.823498821 +0000 UTC m=+976.166447569"
Oct 29 22:59:32.873833 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:32.873378    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:33.281172 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:33.281095    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:33.299284 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:33.298930    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:59:33.299284 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:33.298996    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4" podUID="d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 29 22:59:33.365689 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:33.365582    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:33.630828 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:33.630014    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:34.170832 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:34.169907    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:34.212825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:34.196042    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded"
Oct 29 22:59:34.291838 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:34.287401    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:59:34.291838 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:34.287462    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4" podUID="d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 29 22:59:34.375831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:34.371264    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:59:34.817827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:34.796961    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj" podUID="63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:34.844391 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:34.844338    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 29 22:59:34.890556 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:34.881359    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:34.920848 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:34.911530    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:35.161826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:35.143498    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:35.172272 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:35.172233    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:35.173434 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:35.173409    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:59:35.434254 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:35.434151    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:35.472957 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:35.435025    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:35.519728 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:35.471708    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:59:35.519728 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:35.519335    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4" podUID="d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 29 22:59:35.548870 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:35.531912    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:59:35.671205 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:35.640027    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:35.915827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:35.903183    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj" podUID="63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:59:35.937921 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:35.937879    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 29 22:59:36.036844 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:35.905406    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:36.036844 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:35.942151    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:59:36.036844 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:35.963302    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:59:36.036844 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:35.945691    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:59:36.036844 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:35.964133    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:59:36.110827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:36.064022    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded"
Oct 29 22:59:36.289117 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:36.210429    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 29 22:59:36.324183 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:36.323871    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:36.577882 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:36.542976    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:36.694041 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:36.693992    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:36.723826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:36.707415    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:59:36.723826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:36.707659    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:59:36.725087 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:36.725036    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:59:36.814035 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:36.813973    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4" podUID="d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 29 22:59:36.814673 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:36.776998    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj" podUID="63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": context deadline exceeded"
Oct 29 22:59:36.920932 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:36.920884    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj" podUID="63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:36.985087 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:36.976083    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:37.151826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:37.092965    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:59:37.215379 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:37.117973    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 22:59:37.411826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:37.404876    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 29 22:59:37.539491 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:37.539400    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:59:37.721085 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:37.721033    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:37.816884 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:37.783129    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 29 22:59:38.007821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.007398    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:38.031839 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.025509    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:38.032466 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.032437    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:59:38.032646 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.032623    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4" podUID="d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 29 22:59:38.045894 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.045851    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj" podUID="63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:59:38.230918 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.230871    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:59:38.245305 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.245275    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 22:59:38.271607 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.271563    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:38.320304 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.290017    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:59:38.338397 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.338338    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4" podUID="d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 29 22:59:38.343095 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.330496    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 22:59:38.436842 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.431297    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 22:59:38.436842 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.431353    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 22:59:38.436842 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.431385    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 22:59:38.436842 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.431414    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 22:59:38.436842 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.431455    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 22:59:38.436842 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.420512    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 29 22:59:38.441643 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.441614    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"]
Oct 29 22:59:38.450781 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.450747    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 22:59:38.451089 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.451047    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"
Oct 29 22:59:38.454736 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.454710    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cdxl\" (UniqueName: \"kubernetes.io/projected/0a1a7c9a-d2ea-492a-9324-3596700b59d2-kube-api-access-2cdxl\") pod \"maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j\" (UID: \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"
Oct 29 22:59:38.455021 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.454877    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"]
Oct 29 22:59:38.455329 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.455309    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a1a7c9a-d2ea-492a-9324-3596700b59d2-metrics-certs\") pod \"maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j\" (UID: \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"
Oct 29 22:59:38.455568 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.455551    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a1a7c9a-d2ea-492a-9324-3596700b59d2-openshift-service-ca\") pod \"maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j\" (UID: \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"
Oct 29 22:59:38.455913 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.455788    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a1a7c9a-d2ea-492a-9324-3596700b59d2-maas-api-tls\") pod \"maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j\" (UID: \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"
Oct 29 22:59:38.515907 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.468529    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-479c2a-serving-cert\""
Oct 29 22:59:38.523849 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.523818    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:38.557619 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.557588    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a1a7c9a-d2ea-492a-9324-3596700b59d2-metrics-certs\") pod \"maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j\" (UID: \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"
Oct 29 22:59:38.558571 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.558542    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a1a7c9a-d2ea-492a-9324-3596700b59d2-openshift-service-ca\") pod \"maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j\" (UID: \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"
Oct 29 22:59:38.558697 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.558649    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a1a7c9a-d2ea-492a-9324-3596700b59d2-maas-api-tls\") pod \"maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j\" (UID: \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"
Oct 29 22:59:38.558788 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.558768    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cdxl\" (UniqueName: \"kubernetes.io/projected/0a1a7c9a-d2ea-492a-9324-3596700b59d2-kube-api-access-2cdxl\") pod \"maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j\" (UID: \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"
Oct 29 22:59:38.586831 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.572414    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a1a7c9a-d2ea-492a-9324-3596700b59d2-openshift-service-ca\") pod \"maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j\" (UID: \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"
Oct 29 22:59:38.626792 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.626031    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a1a7c9a-d2ea-492a-9324-3596700b59d2-metrics-certs\") pod \"maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j\" (UID: \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"
Oct 29 22:59:38.638855 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.636670    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cdxl\" (UniqueName: \"kubernetes.io/projected/0a1a7c9a-d2ea-492a-9324-3596700b59d2-kube-api-access-2cdxl\") pod \"maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j\" (UID: \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"
Oct 29 22:59:38.638855 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.636937    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a1a7c9a-d2ea-492a-9324-3596700b59d2-maas-api-tls\") pod \"maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j\" (UID: \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"
Oct 29 22:59:38.848667 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:38.848576    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"
Oct 29 22:59:39.257710 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:39.257587    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:39.301192 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:39.278456    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:59:39.301192 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:39.278520    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4" podUID="d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 29 22:59:39.413518 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:39.413479    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:40.045825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:40.040584    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:59:40.240050 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:40.240005    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:59:40.256085 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:40.256029    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:40.296892 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:40.296319    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 29 22:59:40.296892 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:40.296390    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4" podUID="d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 29 22:59:40.316835 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:40.311464    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:40.412408 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:40.412363    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:40.600833 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:40.595942    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"]
Oct 29 22:59:40.601071 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:40.600846    5428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:59:40.735450 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:40.735420    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr"
Oct 29 22:59:40.767766 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:40.767726    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:41.386178 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:41.386133    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j" event={"ID":"0a1a7c9a-d2ea-492a-9324-3596700b59d2","Type":"ContainerStarted","Data":"f6126a72de5e71e6b6db3c1a5dc6b11224cadf8f4dd70b0eb4f5d325b70a4225"}
Oct 29 22:59:41.386656 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:41.386208    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j" event={"ID":"0a1a7c9a-d2ea-492a-9324-3596700b59d2","Type":"ContainerStarted","Data":"725369ea985b2061252833465e9f53384bff388cc9c6dd0d9a5f479692040888"}
Oct 29 22:59:41.386656 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:41.386240    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"
Oct 29 22:59:41.430218 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:41.430152    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j" podStartSLOduration=3.95572254 podStartE2EDuration="4.430132808s" totalImagesPullingTime="474.410268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:41.42437664 +0000 UTC m=+984.767325384" watchObservedRunningTime="2026-10-29 22:59:41.430132808 +0000 UTC m=+984.773081553"
Oct 29 22:59:41.852586 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:41.852508    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:42.332369 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:42.332319    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:42.339273 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:42.338941    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:59:42.339273 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:42.339005    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4" podUID="d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:43.004222 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:43.004180    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:43.118842 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:43.116884    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:43.229593 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:43.228191    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:43.229593 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:43.228844    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:43.231998 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:43.231942    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 22:59:44.851037 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:44.850995    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:46.276734 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:46.276702    5428 patch_prober.go:29] interesting pod/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:59:46.277180 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:46.276764    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p" podUID="c1804496-00d1-4679-a069-89a1caec4e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:46.581925 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:46.581835    5428 generic.go:356] "Generic (PLEG): container finished" podID="c1804496-00d1-4679-a069-89a1caec4e75" containerID="24d1bd4a76f86cd41d477586472dd4f80b80aced81c3182a213ec406bf54a425" exitCode=0
Oct 29 22:59:46.581925 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:46.581909    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p" event={"ID":"c1804496-00d1-4679-a069-89a1caec4e75","Type":"ContainerDied","Data":"24d1bd4a76f86cd41d477586472dd4f80b80aced81c3182a213ec406bf54a425"}
Oct 29 22:59:46.582373 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:46.582358    5428 scope.go:122] "RemoveContainer" containerID="24d1bd4a76f86cd41d477586472dd4f80b80aced81c3182a213ec406bf54a425"
Oct 29 22:59:46.584793 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:46.584749    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:59:46.584960 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:46.584939    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:59:46.585060 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:46.585043    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:59:46.811296 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:46.811264    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:46.936190 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:46.936155    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:47.369503 ip-10-0-134-127 kubenswrapper[5428]: E1029 22:59:47.369451    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 22:59:47.556833 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:47.556769    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"
Oct 29 22:59:47.595067 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:47.595014    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p" event={"ID":"c1804496-00d1-4679-a069-89a1caec4e75","Type":"ContainerStarted","Data":"91e843f7914e7e8cbd4575f6fbedee8d396604fd3d062ee749133889a608a02d"}
Oct 29 22:59:48.351989 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:48.351958    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:59:48.352226 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:48.352204    5428 patch_prober.go:29] interesting pod/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:59:48.352311 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:48.352270    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p" podUID="c1804496-00d1-4679-a069-89a1caec4e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 29 22:59:48.519148 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:48.519115    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:48.650329 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:48.650298    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:49.155401 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:49.155327    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:49.351854 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:49.351778    5428 patch_prober.go:29] interesting pod/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:59:49.352052 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:49.351900    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p" podUID="c1804496-00d1-4679-a069-89a1caec4e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 29 22:59:50.352361 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:50.352326    5428 patch_prober.go:29] interesting pod/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:59:50.352737 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:50.352379    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p" podUID="c1804496-00d1-4679-a069-89a1caec4e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 29 22:59:50.685190 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:50.685154    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:51.188200 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:51.187289    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:51.302968 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:51.302917    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:51.361824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:51.361435    5428 patch_prober.go:29] interesting pod/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:59:51.361824 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:51.361511    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p" podUID="c1804496-00d1-4679-a069-89a1caec4e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 29 22:59:51.509125 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:51.509026    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:52.356793 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:52.356758    5428 patch_prober.go:29] interesting pod/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:59:52.357295 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:52.357259    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p" podUID="c1804496-00d1-4679-a069-89a1caec4e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 29 22:59:53.316228 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:53.316190    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:53.352305 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:53.352043    5428 patch_prober.go:29] interesting pod/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 29 22:59:53.352305 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:53.352110    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p" podUID="c1804496-00d1-4679-a069-89a1caec4e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 29 22:59:54.355980 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:54.355952    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:59:54.356447 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:54.356424    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:59:54.357146 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:54.357124    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 22:59:55.222971 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:55.222931    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:55.353552 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:55.353512    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:56.864296 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:56.864213    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:56.982714 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:56.982676    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:57.050288 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:57.050255    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v"]
Oct 29 22:59:57.054856 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:57.054831    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v"
Oct 29 22:59:57.074885 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:57.074075    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v"]
Oct 29 22:59:57.077949 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:57.077924    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-df82ee-dockercfg-jn6sq\""
Oct 29 22:59:57.093655 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:57.093617    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-df82ee\""
Oct 29 22:59:57.172875 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:57.170210    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q6dd\" (UniqueName: \"kubernetes.io/projected/a32d5c46-fd89-4d2e-a902-9c3c80abfcb7-kube-api-access-9q6dd\") pod \"payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v\" (UID: \"a32d5c46-fd89-4d2e-a902-9c3c80abfcb7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v"
Oct 29 22:59:57.172875 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:57.170415    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a32d5c46-fd89-4d2e-a902-9c3c80abfcb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v\" (UID: \"a32d5c46-fd89-4d2e-a902-9c3c80abfcb7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v"
Oct 29 22:59:57.390821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:57.386730    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a32d5c46-fd89-4d2e-a902-9c3c80abfcb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v\" (UID: \"a32d5c46-fd89-4d2e-a902-9c3c80abfcb7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v"
Oct 29 22:59:57.390821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:57.387139    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q6dd\" (UniqueName: \"kubernetes.io/projected/a32d5c46-fd89-4d2e-a902-9c3c80abfcb7-kube-api-access-9q6dd\") pod \"payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v\" (UID: \"a32d5c46-fd89-4d2e-a902-9c3c80abfcb7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v"
Oct 29 22:59:57.390821 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:57.388550    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a32d5c46-fd89-4d2e-a902-9c3c80abfcb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v\" (UID: \"a32d5c46-fd89-4d2e-a902-9c3c80abfcb7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v"
Oct 29 22:59:57.437068 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:57.435733    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q6dd\" (UniqueName: \"kubernetes.io/projected/a32d5c46-fd89-4d2e-a902-9c3c80abfcb7-kube-api-access-9q6dd\") pod \"payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v\" (UID: \"a32d5c46-fd89-4d2e-a902-9c3c80abfcb7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v"
Oct 29 22:59:57.640243 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:57.640201    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:57.683633 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:57.682146    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v"
Oct 29 22:59:58.173233 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:59:58.161366    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32d5c46_fd89_4d2e_a902_9c3c80abfcb7.slice/crio-1cf4539ec8f438a4d8350ba55a0e50dcebd58ca072fa5a391d676ef6d8442b87 WatchSource:0}: Error finding container 1cf4539ec8f438a4d8350ba55a0e50dcebd58ca072fa5a391d676ef6d8442b87: Status 404 returned error can't find the container with id 1cf4539ec8f438a4d8350ba55a0e50dcebd58ca072fa5a391d676ef6d8442b87
Oct 29 22:59:58.178314 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:58.175246    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v"]
Oct 29 22:59:58.412764 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:58.412714    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz"]
Oct 29 22:59:58.651039 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:58.649295    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:58.768375 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:58.767356    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:58.821448 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:58.819732    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v"
Oct 29 22:59:58.821448 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:58.819793    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v" event={"ID":"a32d5c46-fd89-4d2e-a902-9c3c80abfcb7","Type":"ContainerStarted","Data":"844e1c998df1584e86fbf9725b96ed740da2b51d570e88c3108227c88fddd6d5"}
Oct 29 22:59:58.821448 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:58.819842    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v" event={"ID":"a32d5c46-fd89-4d2e-a902-9c3c80abfcb7","Type":"ContainerStarted","Data":"1cf4539ec8f438a4d8350ba55a0e50dcebd58ca072fa5a391d676ef6d8442b87"}
Oct 29 22:59:58.827872 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:58.825207    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v" podStartSLOduration=1.825187814 podStartE2EDuration="1.825187814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:58.823109162 +0000 UTC m=+1002.166057908" watchObservedRunningTime="2026-10-29 22:59:58.825187814 +0000 UTC m=+1002.168136560"
Oct 29 22:59:58.890672 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:58.890627    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"]
Oct 29 22:59:58.894567 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:58.894544    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:58.906718 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:58.906688    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"]
Oct 29 22:59:58.914362 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:58.914335    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7dsjj\""
Oct 29 22:59:58.914521 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:58.914407    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 29 22:59:58.914565 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:58.914547    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 29 22:59:58.914620 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:58.914607    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 29 22:59:59.036425 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.036344    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.036425 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.036382    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.036626 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.036481    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.036626 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.036526    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh4cn\" (UniqueName: \"kubernetes.io/projected/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-kube-api-access-xh4cn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.036626 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.036602    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.036779 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.036765    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.036941 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.036928    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.157825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.139855    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.157825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.140025    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.157825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.140177    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.157825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.140201    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.157825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.140251    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.157825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.140286    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh4cn\" (UniqueName: \"kubernetes.io/projected/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-kube-api-access-xh4cn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.157825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.140346    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.157825 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.141308    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.245827 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.245284    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.246304 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.246180    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.274390 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.274353    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.289870 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.288817    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh4cn\" (UniqueName: \"kubernetes.io/projected/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-kube-api-access-xh4cn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.289870 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.289486    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.335885 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.333739    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vs95c\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.517725 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.517677    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 22:59:59.782826 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.780691    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:59.890960 ip-10-0-134-127 kubenswrapper[5428]: W1029 22:59:59.890929    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3c6c27_9aa7_4242_b230_23d1ce4a2dcb.slice/crio-cf2a9e7e795b3ddeb49981b8321d1bc138d4e41ddd5bd170cfc890d59fd6406a WatchSource:0}: Error finding container cf2a9e7e795b3ddeb49981b8321d1bc138d4e41ddd5bd170cfc890d59fd6406a: Status 404 returned error can't find the container with id cf2a9e7e795b3ddeb49981b8321d1bc138d4e41ddd5bd170cfc890d59fd6406a
Oct 29 22:59:59.893999 ip-10-0-134-127 kubenswrapper[5428]: I1029 22:59:59.893976    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"]
Oct 29 23:00:00.196958 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.196872    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw"]
Oct 29 23:00:00.209263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.209231    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz"]
Oct 29 23:00:00.212260 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.212229    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw"
Oct 29 23:00:00.248193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.248122    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw"]
Oct 29 23:00:00.248193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.248181    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k"]
Oct 29 23:00:00.248698 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.248362    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz"
Oct 29 23:00:00.257032 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.256999    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4"]
Oct 29 23:00:00.257617 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.257375    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k"
Oct 29 23:00:00.306134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.306092    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\""
Oct 29 23:00:00.321783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.321755    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz"]
Oct 29 23:00:00.322344 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.322012    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4"
Oct 29 23:00:00.322458 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.322384    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k"]
Oct 29 23:00:00.322458 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.322439    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4"]
Oct 29 23:00:00.403683 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.403649    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqqqw\" (UniqueName: \"kubernetes.io/projected/8647546d-a3ef-427a-9942-2da7c19c3e08-kube-api-access-sqqqw\") pod \"maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k\" (UID: \"8647546d-a3ef-427a-9942-2da7c19c3e08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k"
Oct 29 23:00:00.403850 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.403785    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63c04a5-4ccd-4e8c-8dee-ec131b926136-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw\" (UID: \"e63c04a5-4ccd-4e8c-8dee-ec131b926136\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw"
Oct 29 23:00:00.403939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.403912    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qxjz\" (UniqueName: \"kubernetes.io/projected/e63c04a5-4ccd-4e8c-8dee-ec131b926136-kube-api-access-9qxjz\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw\" (UID: \"e63c04a5-4ccd-4e8c-8dee-ec131b926136\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw"
Oct 29 23:00:00.403980 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.403959    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmcfb\" (UniqueName: \"kubernetes.io/projected/e9d05b45-9d7e-424b-9313-c1551ceb45a3-kube-api-access-hmcfb\") pod \"maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz\" (UID: \"e9d05b45-9d7e-424b-9313-c1551ceb45a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz"
Oct 29 23:00:00.404015 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.403982    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d05b45-9d7e-424b-9313-c1551ceb45a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz\" (UID: \"e9d05b45-9d7e-424b-9313-c1551ceb45a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz"
Oct 29 23:00:00.404117 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.404106    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8647546d-a3ef-427a-9942-2da7c19c3e08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k\" (UID: \"8647546d-a3ef-427a-9942-2da7c19c3e08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k"
Oct 29 23:00:00.506041 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.505870    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqqqw\" (UniqueName: \"kubernetes.io/projected/8647546d-a3ef-427a-9942-2da7c19c3e08-kube-api-access-sqqqw\") pod \"maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k\" (UID: \"8647546d-a3ef-427a-9942-2da7c19c3e08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k"
Oct 29 23:00:00.506453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.506411    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63c04a5-4ccd-4e8c-8dee-ec131b926136-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw\" (UID: \"e63c04a5-4ccd-4e8c-8dee-ec131b926136\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw"
Oct 29 23:00:00.506588 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.506505    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qxjz\" (UniqueName: \"kubernetes.io/projected/e63c04a5-4ccd-4e8c-8dee-ec131b926136-kube-api-access-9qxjz\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw\" (UID: \"e63c04a5-4ccd-4e8c-8dee-ec131b926136\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw"
Oct 29 23:00:00.506588 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.506542    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmcfb\" (UniqueName: \"kubernetes.io/projected/e9d05b45-9d7e-424b-9313-c1551ceb45a3-kube-api-access-hmcfb\") pod \"maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz\" (UID: \"e9d05b45-9d7e-424b-9313-c1551ceb45a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz"
Oct 29 23:00:00.506588 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.506578    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d05b45-9d7e-424b-9313-c1551ceb45a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz\" (UID: \"e9d05b45-9d7e-424b-9313-c1551ceb45a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz"
Oct 29 23:00:00.506793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.506733    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8647546d-a3ef-427a-9942-2da7c19c3e08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k\" (UID: \"8647546d-a3ef-427a-9942-2da7c19c3e08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k"
Oct 29 23:00:00.506793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.506775    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcx5t\" (UniqueName: \"kubernetes.io/projected/080a2bc6-16d5-49ca-88c6-a41225b85c57-kube-api-access-wcx5t\") pod \"maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4\" (UID: \"080a2bc6-16d5-49ca-88c6-a41225b85c57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4"
Oct 29 23:00:00.506967 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.506946    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/080a2bc6-16d5-49ca-88c6-a41225b85c57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4\" (UID: \"080a2bc6-16d5-49ca-88c6-a41225b85c57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4"
Oct 29 23:00:00.507176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.507151    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63c04a5-4ccd-4e8c-8dee-ec131b926136-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw\" (UID: \"e63c04a5-4ccd-4e8c-8dee-ec131b926136\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw"
Oct 29 23:00:00.507381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.507361    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8647546d-a3ef-427a-9942-2da7c19c3e08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k\" (UID: \"8647546d-a3ef-427a-9942-2da7c19c3e08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k"
Oct 29 23:00:00.507449 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.507411    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d05b45-9d7e-424b-9313-c1551ceb45a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz\" (UID: \"e9d05b45-9d7e-424b-9313-c1551ceb45a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz"
Oct 29 23:00:00.528497 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.528468    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qxjz\" (UniqueName: \"kubernetes.io/projected/e63c04a5-4ccd-4e8c-8dee-ec131b926136-kube-api-access-9qxjz\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw\" (UID: \"e63c04a5-4ccd-4e8c-8dee-ec131b926136\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw"
Oct 29 23:00:00.528600 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.528471    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmcfb\" (UniqueName: \"kubernetes.io/projected/e9d05b45-9d7e-424b-9313-c1551ceb45a3-kube-api-access-hmcfb\") pod \"maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz\" (UID: \"e9d05b45-9d7e-424b-9313-c1551ceb45a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz"
Oct 29 23:00:00.528677 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.528658    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqqqw\" (UniqueName: \"kubernetes.io/projected/8647546d-a3ef-427a-9942-2da7c19c3e08-kube-api-access-sqqqw\") pod \"maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k\" (UID: \"8647546d-a3ef-427a-9942-2da7c19c3e08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k"
Oct 29 23:00:00.549668 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.549641    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw"
Oct 29 23:00:00.609458 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.609420    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcx5t\" (UniqueName: \"kubernetes.io/projected/080a2bc6-16d5-49ca-88c6-a41225b85c57-kube-api-access-wcx5t\") pod \"maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4\" (UID: \"080a2bc6-16d5-49ca-88c6-a41225b85c57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4"
Oct 29 23:00:00.609617 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.609555    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/080a2bc6-16d5-49ca-88c6-a41225b85c57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4\" (UID: \"080a2bc6-16d5-49ca-88c6-a41225b85c57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4"
Oct 29 23:00:00.610300 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.610267    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/080a2bc6-16d5-49ca-88c6-a41225b85c57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4\" (UID: \"080a2bc6-16d5-49ca-88c6-a41225b85c57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4"
Oct 29 23:00:00.623837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.623797    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz"
Oct 29 23:00:00.630393 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.630364    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k"
Oct 29 23:00:00.631639 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.631609    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcx5t\" (UniqueName: \"kubernetes.io/projected/080a2bc6-16d5-49ca-88c6-a41225b85c57-kube-api-access-wcx5t\") pod \"maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4\" (UID: \"080a2bc6-16d5-49ca-88c6-a41225b85c57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4"
Oct 29 23:00:00.641071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.641044    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4"
Oct 29 23:00:00.831379 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.831341    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c" event={"ID":"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb","Type":"ContainerStarted","Data":"cf2a9e7e795b3ddeb49981b8321d1bc138d4e41ddd5bd170cfc890d59fd6406a"}
Oct 29 23:00:00.903291 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:00.903259    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw"]
Oct 29 23:00:00.906475 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:00:00.906450    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63c04a5_4ccd_4e8c_8dee_ec131b926136.slice/crio-02e96d8eb85418df39cf6c786f56280b42ef9761c1f4ecff650082b1446b4d0b WatchSource:0}: Error finding container 02e96d8eb85418df39cf6c786f56280b42ef9761c1f4ecff650082b1446b4d0b: Status 404 returned error can't find the container with id 02e96d8eb85418df39cf6c786f56280b42ef9761c1f4ecff650082b1446b4d0b
Oct 29 23:00:01.041382 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:01.041313    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz"]
Oct 29 23:00:01.047498 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:00:01.047467    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d05b45_9d7e_424b_9313_c1551ceb45a3.slice/crio-96c7927fcf2b9c7d8ac3a598b18c14ec43c54dda6de858d169d22f2145bce597 WatchSource:0}: Error finding container 96c7927fcf2b9c7d8ac3a598b18c14ec43c54dda6de858d169d22f2145bce597: Status 404 returned error can't find the container with id 96c7927fcf2b9c7d8ac3a598b18c14ec43c54dda6de858d169d22f2145bce597
Oct 29 23:00:01.101915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:01.101791    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4"]
Oct 29 23:00:01.107070 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:00:01.107034    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080a2bc6_16d5_49ca_88c6_a41225b85c57.slice/crio-48ee26afeed41d0f8d35b5534bde5e68d2a48ebf216f03aa917a070c9c817d69 WatchSource:0}: Error finding container 48ee26afeed41d0f8d35b5534bde5e68d2a48ebf216f03aa917a070c9c817d69: Status 404 returned error can't find the container with id 48ee26afeed41d0f8d35b5534bde5e68d2a48ebf216f03aa917a070c9c817d69
Oct 29 23:00:01.108850 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:00:01.108814    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8647546d_a3ef_427a_9942_2da7c19c3e08.slice/crio-f54578205cf40e61f69fc56ecaed7c57a48631716dda5b6bc2ab9e389bc2d37d WatchSource:0}: Error finding container f54578205cf40e61f69fc56ecaed7c57a48631716dda5b6bc2ab9e389bc2d37d: Status 404 returned error can't find the container with id f54578205cf40e61f69fc56ecaed7c57a48631716dda5b6bc2ab9e389bc2d37d
Oct 29 23:00:01.112400 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:01.112187    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k"]
Oct 29 23:00:01.585679 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:01.585627    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:01.845354 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:01.845251    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz" event={"ID":"e9d05b45-9d7e-424b-9313-c1551ceb45a3","Type":"ContainerStarted","Data":"dfb30a572db3e0eee05bc6da4dc342ef5934b153bdf148ab75b470605e2811c8"}
Oct 29 23:00:01.845354 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:01.845325    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz" event={"ID":"e9d05b45-9d7e-424b-9313-c1551ceb45a3","Type":"ContainerStarted","Data":"96c7927fcf2b9c7d8ac3a598b18c14ec43c54dda6de858d169d22f2145bce597"}
Oct 29 23:00:01.848079 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:01.848051    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4" event={"ID":"080a2bc6-16d5-49ca-88c6-a41225b85c57","Type":"ContainerStarted","Data":"1e9d8a6973398e681d3289bb5cdeffe9bbcde48e893783462310d29661894248"}
Oct 29 23:00:01.848238 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:01.848093    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4" event={"ID":"080a2bc6-16d5-49ca-88c6-a41225b85c57","Type":"ContainerStarted","Data":"48ee26afeed41d0f8d35b5534bde5e68d2a48ebf216f03aa917a070c9c817d69"}
Oct 29 23:00:01.854825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:01.854779    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw" event={"ID":"e63c04a5-4ccd-4e8c-8dee-ec131b926136","Type":"ContainerStarted","Data":"946c32914ef1915ce598022d49990d86eab781e2fd7592fb4b271c5188882c21"}
Oct 29 23:00:01.854959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:01.854848    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw" event={"ID":"e63c04a5-4ccd-4e8c-8dee-ec131b926136","Type":"ContainerStarted","Data":"02e96d8eb85418df39cf6c786f56280b42ef9761c1f4ecff650082b1446b4d0b"}
Oct 29 23:00:01.858062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:01.858031    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k" event={"ID":"8647546d-a3ef-427a-9942-2da7c19c3e08","Type":"ContainerStarted","Data":"02fe20238d3820dcde7506d94827fc85c47e615968c51c11aa43ef046dd48874"}
Oct 29 23:00:01.858185 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:01.858080    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k" event={"ID":"8647546d-a3ef-427a-9942-2da7c19c3e08","Type":"ContainerStarted","Data":"f54578205cf40e61f69fc56ecaed7c57a48631716dda5b6bc2ab9e389bc2d37d"}
Oct 29 23:00:01.909239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:01.909180    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz" podStartSLOduration=1.909165458 podStartE2EDuration="1.909165458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:01.907085467 +0000 UTC m=+1005.250034230" watchObservedRunningTime="2026-10-29 23:00:01.909165458 +0000 UTC m=+1005.252114247"
Oct 29 23:00:01.945391 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:01.945341    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4" podStartSLOduration=1.945326796 podStartE2EDuration="1.945326796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:01.942638681 +0000 UTC m=+1005.285587426" watchObservedRunningTime="2026-10-29 23:00:01.945326796 +0000 UTC m=+1005.288275541"
Oct 29 23:00:02.048701 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:02.048635    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw" podStartSLOduration=2.048615196 podStartE2EDuration="2.048615196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.009040068 +0000 UTC m=+1005.351988814" watchObservedRunningTime="2026-10-29 23:00:02.048615196 +0000 UTC m=+1005.391563942"
Oct 29 23:00:02.052296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:02.052254    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k" podStartSLOduration=2.052241574 podStartE2EDuration="2.052241574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.044540528 +0000 UTC m=+1005.387489285" watchObservedRunningTime="2026-10-29 23:00:02.052241574 +0000 UTC m=+1005.395190345"
Oct 29 23:00:02.894717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:02.894679    5428 generic.go:356] "Generic (PLEG): container finished" podID="8647546d-a3ef-427a-9942-2da7c19c3e08" containerID="02fe20238d3820dcde7506d94827fc85c47e615968c51c11aa43ef046dd48874" exitCode=60
Oct 29 23:00:02.895429 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:02.894851    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k" event={"ID":"8647546d-a3ef-427a-9942-2da7c19c3e08","Type":"ContainerDied","Data":"02fe20238d3820dcde7506d94827fc85c47e615968c51c11aa43ef046dd48874"}
Oct 29 23:00:02.895509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:02.895486    5428 scope.go:122] "RemoveContainer" containerID="02fe20238d3820dcde7506d94827fc85c47e615968c51c11aa43ef046dd48874"
Oct 29 23:00:02.899433 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:02.899411    5428 generic.go:356] "Generic (PLEG): container finished" podID="e9d05b45-9d7e-424b-9313-c1551ceb45a3" containerID="dfb30a572db3e0eee05bc6da4dc342ef5934b153bdf148ab75b470605e2811c8" exitCode=60
Oct 29 23:00:02.899921 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:02.899896    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz" event={"ID":"e9d05b45-9d7e-424b-9313-c1551ceb45a3","Type":"ContainerDied","Data":"dfb30a572db3e0eee05bc6da4dc342ef5934b153bdf148ab75b470605e2811c8"}
Oct 29 23:00:02.900564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:02.900546    5428 scope.go:122] "RemoveContainer" containerID="dfb30a572db3e0eee05bc6da4dc342ef5934b153bdf148ab75b470605e2811c8"
Oct 29 23:00:02.916364 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:02.916338    5428 generic.go:356] "Generic (PLEG): container finished" podID="080a2bc6-16d5-49ca-88c6-a41225b85c57" containerID="1e9d8a6973398e681d3289bb5cdeffe9bbcde48e893783462310d29661894248" exitCode=60
Oct 29 23:00:02.916479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:02.916407    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4" event={"ID":"080a2bc6-16d5-49ca-88c6-a41225b85c57","Type":"ContainerDied","Data":"1e9d8a6973398e681d3289bb5cdeffe9bbcde48e893783462310d29661894248"}
Oct 29 23:00:02.916881 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:02.916861    5428 scope.go:122] "RemoveContainer" containerID="1e9d8a6973398e681d3289bb5cdeffe9bbcde48e893783462310d29661894248"
Oct 29 23:00:02.919645 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:02.919621    5428 generic.go:356] "Generic (PLEG): container finished" podID="e63c04a5-4ccd-4e8c-8dee-ec131b926136" containerID="946c32914ef1915ce598022d49990d86eab781e2fd7592fb4b271c5188882c21" exitCode=60
Oct 29 23:00:02.919751 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:02.919693    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw" event={"ID":"e63c04a5-4ccd-4e8c-8dee-ec131b926136","Type":"ContainerDied","Data":"946c32914ef1915ce598022d49990d86eab781e2fd7592fb4b271c5188882c21"}
Oct 29 23:00:02.920048 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:02.920029    5428 scope.go:122] "RemoveContainer" containerID="946c32914ef1915ce598022d49990d86eab781e2fd7592fb4b271c5188882c21"
Oct 29 23:00:03.349122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:03.349083    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:03.413700 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:03.413659    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz" podUID="3982f117-51f0-44b5-aa87-a5a6f3132fab" containerName="payload-pre-processing" containerID="cri-o://fb497befbae7bcbfc29c081478d54251e1d8a2433f1deb9acf602901a3ffcf3f" gracePeriod=25
Oct 29 23:00:03.931038 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:03.931004    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4" event={"ID":"080a2bc6-16d5-49ca-88c6-a41225b85c57","Type":"ContainerStarted","Data":"b4b950539f2ed132b89a4dbbce268c9edd618a01d9c92c5dfa638c9e52d46b48"}
Oct 29 23:00:03.933685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:03.933657    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw" event={"ID":"e63c04a5-4ccd-4e8c-8dee-ec131b926136","Type":"ContainerStarted","Data":"29d7538e34dcf564a621ec01f65e913c365032d341b97abfec679f83ad13cffc"}
Oct 29 23:00:03.936361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:03.936336    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k" event={"ID":"8647546d-a3ef-427a-9942-2da7c19c3e08","Type":"ContainerStarted","Data":"b9e949f20f64d79cc880865572ffb107516a5694acb8ca393974b98d9e04bbf8"}
Oct 29 23:00:03.939593 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:03.939569    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz" event={"ID":"e9d05b45-9d7e-424b-9313-c1551ceb45a3","Type":"ContainerStarted","Data":"35bbeb63d44a72c5c5d2aea2d166e1eb673a150c957c4f68dbf5a5487fd8a303"}
Oct 29 23:00:04.054701 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:04.054666    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:00:04.054876 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:04.054743    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:00:04.991160 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:04.991124    5428 generic.go:356] "Generic (PLEG): container finished" podID="e9d05b45-9d7e-424b-9313-c1551ceb45a3" containerID="35bbeb63d44a72c5c5d2aea2d166e1eb673a150c957c4f68dbf5a5487fd8a303" exitCode=60
Oct 29 23:00:04.991715 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:04.991344    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz" event={"ID":"e9d05b45-9d7e-424b-9313-c1551ceb45a3","Type":"ContainerDied","Data":"35bbeb63d44a72c5c5d2aea2d166e1eb673a150c957c4f68dbf5a5487fd8a303"}
Oct 29 23:00:04.991715 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:04.991404    5428 scope.go:122] "RemoveContainer" containerID="dfb30a572db3e0eee05bc6da4dc342ef5934b153bdf148ab75b470605e2811c8"
Oct 29 23:00:04.991715 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:04.991649    5428 scope.go:122] "RemoveContainer" containerID="35bbeb63d44a72c5c5d2aea2d166e1eb673a150c957c4f68dbf5a5487fd8a303"
Oct 29 23:00:04.992236 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:00:04.992203    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz_odh-ai-gateway-infra(e9d05b45-9d7e-424b-9313-c1551ceb45a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz" podUID="e9d05b45-9d7e-424b-9313-c1551ceb45a3"
Oct 29 23:00:05.016853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:05.016739    5428 generic.go:356] "Generic (PLEG): container finished" podID="080a2bc6-16d5-49ca-88c6-a41225b85c57" containerID="b4b950539f2ed132b89a4dbbce268c9edd618a01d9c92c5dfa638c9e52d46b48" exitCode=60
Oct 29 23:00:05.017570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:05.017121    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4" event={"ID":"080a2bc6-16d5-49ca-88c6-a41225b85c57","Type":"ContainerDied","Data":"b4b950539f2ed132b89a4dbbce268c9edd618a01d9c92c5dfa638c9e52d46b48"}
Oct 29 23:00:05.017828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:05.017235    5428 scope.go:122] "RemoveContainer" containerID="b4b950539f2ed132b89a4dbbce268c9edd618a01d9c92c5dfa638c9e52d46b48"
Oct 29 23:00:05.018308 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:00:05.018280    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4_odh-ai-gateway-infra(080a2bc6-16d5-49ca-88c6-a41225b85c57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4" podUID="080a2bc6-16d5-49ca-88c6-a41225b85c57"
Oct 29 23:00:05.050862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:05.031633    5428 generic.go:356] "Generic (PLEG): container finished" podID="e63c04a5-4ccd-4e8c-8dee-ec131b926136" containerID="29d7538e34dcf564a621ec01f65e913c365032d341b97abfec679f83ad13cffc" exitCode=60
Oct 29 23:00:05.050862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:05.031881    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw" event={"ID":"e63c04a5-4ccd-4e8c-8dee-ec131b926136","Type":"ContainerDied","Data":"29d7538e34dcf564a621ec01f65e913c365032d341b97abfec679f83ad13cffc"}
Oct 29 23:00:05.050862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:05.032630    5428 scope.go:122] "RemoveContainer" containerID="29d7538e34dcf564a621ec01f65e913c365032d341b97abfec679f83ad13cffc"
Oct 29 23:00:05.050862 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:00:05.033729    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw_odh-ai-gateway-infra(e63c04a5-4ccd-4e8c-8dee-ec131b926136)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw" podUID="e63c04a5-4ccd-4e8c-8dee-ec131b926136"
Oct 29 23:00:05.050862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:05.048468    5428 scope.go:122] "RemoveContainer" containerID="1e9d8a6973398e681d3289bb5cdeffe9bbcde48e893783462310d29661894248"
Oct 29 23:00:05.050862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:05.049498    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c" event={"ID":"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb","Type":"ContainerStarted","Data":"3d60f1706b8bafeb32527bdeaf2daa9a5edf877c16309c83e8fedcb40f8d88e8"}
Oct 29 23:00:05.051322 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:05.050912    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 23:00:05.063154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:05.063120    5428 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vs95c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:00:05.064544 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:05.064520    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c" podUID="7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:00:05.065895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:05.065873    5428 generic.go:356] "Generic (PLEG): container finished" podID="8647546d-a3ef-427a-9942-2da7c19c3e08" containerID="b9e949f20f64d79cc880865572ffb107516a5694acb8ca393974b98d9e04bbf8" exitCode=60
Oct 29 23:00:05.066135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:05.066097    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k" event={"ID":"8647546d-a3ef-427a-9942-2da7c19c3e08","Type":"ContainerDied","Data":"b9e949f20f64d79cc880865572ffb107516a5694acb8ca393974b98d9e04bbf8"}
Oct 29 23:00:05.066670 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:05.066653    5428 scope.go:122] "RemoveContainer" containerID="b9e949f20f64d79cc880865572ffb107516a5694acb8ca393974b98d9e04bbf8"
Oct 29 23:00:05.067281 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:00:05.067260    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k_odh-ai-gateway-infra(8647546d-a3ef-427a-9942-2da7c19c3e08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k" podUID="8647546d-a3ef-427a-9942-2da7c19c3e08"
Oct 29 23:00:05.090860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:05.089953    5428 scope.go:122] "RemoveContainer" containerID="946c32914ef1915ce598022d49990d86eab781e2fd7592fb4b271c5188882c21"
Oct 29 23:00:05.147567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:05.147524    5428 scope.go:122] "RemoveContainer" containerID="02fe20238d3820dcde7506d94827fc85c47e615968c51c11aa43ef046dd48874"
Oct 29 23:00:05.224452 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:05.224395    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c" podStartSLOduration=3.063920624 podStartE2EDuration="7.224380897s" totalImagesPullingTime="4.160460273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:05.178151524 +0000 UTC m=+1008.521100270" watchObservedRunningTime="2026-10-29 23:00:05.224380897 +0000 UTC m=+1008.567329642"
Oct 29 23:00:05.260867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:05.258667    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:06.078454 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:06.078420    5428 scope.go:122] "RemoveContainer" containerID="b9e949f20f64d79cc880865572ffb107516a5694acb8ca393974b98d9e04bbf8"
Oct 29 23:00:06.079003 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:00:06.078926    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k_odh-ai-gateway-infra(8647546d-a3ef-427a-9942-2da7c19c3e08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k" podUID="8647546d-a3ef-427a-9942-2da7c19c3e08"
Oct 29 23:00:06.080521 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:06.080499    5428 scope.go:122] "RemoveContainer" containerID="35bbeb63d44a72c5c5d2aea2d166e1eb673a150c957c4f68dbf5a5487fd8a303"
Oct 29 23:00:06.080949 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:00:06.080923    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz_odh-ai-gateway-infra(e9d05b45-9d7e-424b-9313-c1551ceb45a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz" podUID="e9d05b45-9d7e-424b-9313-c1551ceb45a3"
Oct 29 23:00:06.082925 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:06.082898    5428 scope.go:122] "RemoveContainer" containerID="b4b950539f2ed132b89a4dbbce268c9edd618a01d9c92c5dfa638c9e52d46b48"
Oct 29 23:00:06.083222 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:00:06.083196    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4_odh-ai-gateway-infra(080a2bc6-16d5-49ca-88c6-a41225b85c57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4" podUID="080a2bc6-16d5-49ca-88c6-a41225b85c57"
Oct 29 23:00:06.085571 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:06.085549    5428 scope.go:122] "RemoveContainer" containerID="29d7538e34dcf564a621ec01f65e913c365032d341b97abfec679f83ad13cffc"
Oct 29 23:00:06.086033 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:00:06.085951    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw_odh-ai-gateway-infra(e63c04a5-4ccd-4e8c-8dee-ec131b926136)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw" podUID="e63c04a5-4ccd-4e8c-8dee-ec131b926136"
Oct 29 23:00:06.086139 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:06.086092    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 23:00:06.779944 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:06.779913    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv"]
Oct 29 23:00:06.849374 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:06.849333    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv"]
Oct 29 23:00:06.849374 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:06.849470    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv"
Oct 29 23:00:06.862667 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:06.862644    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:00:07.006386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:07.006349    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4372d16f-tmxwv\" (UID: \"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv"
Oct 29 23:00:07.006567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:07.006475    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:07.006705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:07.006678    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4372d16f-tmxwv\" (UID: \"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv"
Oct 29 23:00:07.006762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:07.006736    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6nvq\" (UniqueName: \"kubernetes.io/projected/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-kube-api-access-v6nvq\") pod \"maas-api-revoke-sub-4372d16f-tmxwv\" (UID: \"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv"
Oct 29 23:00:07.108419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:07.108328    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4372d16f-tmxwv\" (UID: \"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv"
Oct 29 23:00:07.109946 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:07.108605    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4372d16f-tmxwv\" (UID: \"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv"
Oct 29 23:00:07.109946 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:07.108661    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6nvq\" (UniqueName: \"kubernetes.io/projected/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-kube-api-access-v6nvq\") pod \"maas-api-revoke-sub-4372d16f-tmxwv\" (UID: \"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv"
Oct 29 23:00:07.109946 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:07.109621    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4372d16f-tmxwv\" (UID: \"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv"
Oct 29 23:00:07.178739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:07.178703    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:07.184897 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:07.184867    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4372d16f-tmxwv\" (UID: \"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv"
Oct 29 23:00:07.185260 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:07.185237    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6nvq\" (UniqueName: \"kubernetes.io/projected/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-kube-api-access-v6nvq\") pod \"maas-api-revoke-sub-4372d16f-tmxwv\" (UID: \"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv"
Oct 29 23:00:07.190509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:07.190484    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv"
Oct 29 23:00:07.731740 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:07.731692    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:07.998487 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:07.998458    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv"]
Oct 29 23:00:07.999623 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:00:07.999594    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a434fd6_4f8e_4a9d_a7c8_389a718a45f8.slice/crio-1b8bbda0f79330be6a48d7d0d0e3bc2931f1cdc09c7b5da04588b6045511b00f WatchSource:0}: Error finding container 1b8bbda0f79330be6a48d7d0d0e3bc2931f1cdc09c7b5da04588b6045511b00f: Status 404 returned error can't find the container with id 1b8bbda0f79330be6a48d7d0d0e3bc2931f1cdc09c7b5da04588b6045511b00f
Oct 29 23:00:08.111206 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:08.111170    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv" event={"ID":"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8","Type":"ContainerStarted","Data":"1b8bbda0f79330be6a48d7d0d0e3bc2931f1cdc09c7b5da04588b6045511b00f"}
Oct 29 23:00:08.201428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:08.201397    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:08.484866 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:08.475353    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:09.359184 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:09.359095    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:09.483190 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:09.483023    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:09.703431 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:09.703388    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz" podUID="3982f117-51f0-44b5-aa87-a5a6f3132fab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:09.798380 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:09.798300    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v"
Oct 29 23:00:09.829246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:09.829211    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:10.133581 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:10.133538    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv" event={"ID":"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8","Type":"ContainerStarted","Data":"878f7d3c6c2646f3c7b3e350c7219287431b0e6fa8f19435433c51ced6a444eb"}
Oct 29 23:00:10.173757 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:10.173697    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv" podStartSLOduration=4.173680942 podStartE2EDuration="4.173680942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:10.169175507 +0000 UTC m=+1013.512124252" watchObservedRunningTime="2026-10-29 23:00:10.173680942 +0000 UTC m=+1013.516629686"
Oct 29 23:00:11.143880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:11.143850    5428 generic.go:356] "Generic (PLEG): container finished" podID="1a434fd6-4f8e-4a9d-a7c8-389a718a45f8" containerID="878f7d3c6c2646f3c7b3e350c7219287431b0e6fa8f19435433c51ced6a444eb" exitCode=0
Oct 29 23:00:11.144292 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:11.143987    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv" event={"ID":"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8","Type":"ContainerDied","Data":"878f7d3c6c2646f3c7b3e350c7219287431b0e6fa8f19435433c51ced6a444eb"}
Oct 29 23:00:11.484497 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:11.484462    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:11.741965 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:11.741881    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:12.431549 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:12.431480    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv"
Oct 29 23:00:12.596219 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:12.596186    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-kube-api-access-v6nvq\" (UniqueName: \"kubernetes.io/projected/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-kube-api-access-v6nvq\") pod \"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8\" (UID: \"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8\") "
Oct 29 23:00:12.596489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:12.596273    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-maas-api-openshift-ca\") pod \"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8\" (UID: \"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8\") "
Oct 29 23:00:12.596623 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:12.596608    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-maas-api-xks-ca\") pod \"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8\" (UID: \"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8\") "
Oct 29 23:00:12.614212 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:12.614176    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-maas-api-openshift-ca" pod "1a434fd6-4f8e-4a9d-a7c8-389a718a45f8" (UID: "1a434fd6-4f8e-4a9d-a7c8-389a718a45f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:12.620719 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:12.620693    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-kube-api-access-v6nvq" pod "1a434fd6-4f8e-4a9d-a7c8-389a718a45f8" (UID: "1a434fd6-4f8e-4a9d-a7c8-389a718a45f8"). InnerVolumeSpecName "kube-api-access-v6nvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:12.620856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:12.620759    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-maas-api-xks-ca" pod "1a434fd6-4f8e-4a9d-a7c8-389a718a45f8" (UID: "1a434fd6-4f8e-4a9d-a7c8-389a718a45f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:12.698719 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:12.698630    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-maas-api-xks-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:00:12.698719 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:12.698663    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6nvq\" (UniqueName: \"kubernetes.io/projected/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-kube-api-access-v6nvq\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:00:12.698719 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:12.698674    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8-maas-api-openshift-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.163564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:13.163486    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv" event={"ID":"1a434fd6-4f8e-4a9d-a7c8-389a718a45f8","Type":"ContainerDied","Data":"1b8bbda0f79330be6a48d7d0d0e3bc2931f1cdc09c7b5da04588b6045511b00f"}
Oct 29 23:00:13.163564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:13.163511    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv"
Oct 29 23:00:13.194455 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:13.163541    5428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b8bbda0f79330be6a48d7d0d0e3bc2931f1cdc09c7b5da04588b6045511b00f"
Oct 29 23:00:13.238644 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:13.238611    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:13.351080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:13.351054    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:13.660827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:13.655475    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:14.173536 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:14.173504    5428 generic.go:356] "Generic (PLEG): container finished" podID="d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" containerID="27660b71d8a1de8e19b5f7b9b3431863b594c9aaa8e92e8fcf2aaef13c9bf66b" exitCode=0
Oct 29 23:00:14.173729 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:14.173583    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9" event={"ID":"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763","Type":"ContainerDied","Data":"27660b71d8a1de8e19b5f7b9b3431863b594c9aaa8e92e8fcf2aaef13c9bf66b"}
Oct 29 23:00:14.174122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:14.174095    5428 scope.go:122] "RemoveContainer" containerID="27660b71d8a1de8e19b5f7b9b3431863b594c9aaa8e92e8fcf2aaef13c9bf66b"
Oct 29 23:00:14.175779 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:14.175746    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:00:14.175878 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:14.175865    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:00:14.175934 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:14.175904    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:00:15.200253 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:15.200208    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9" event={"ID":"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763","Type":"ContainerStarted","Data":"9de25fb15a301b9dd0b813455139e1c5037e612081d5f9798343ec99110148eb"}
Oct 29 23:00:15.601312 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:15.601227    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 23:00:15.707228 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:15.707180    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 23:00:15.708389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:15.708362    5428 patch_prober.go:29] interesting pod/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 29 23:00:15.708515 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:15.708437    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9" podUID="d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 29 23:00:15.778242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:15.778210    5428 scope.go:122] "RemoveContainer" containerID="b4b950539f2ed132b89a4dbbce268c9edd618a01d9c92c5dfa638c9e52d46b48"
Oct 29 23:00:15.789027 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:15.788990    5428 scope.go:122] "RemoveContainer" containerID="29d7538e34dcf564a621ec01f65e913c365032d341b97abfec679f83ad13cffc"
Oct 29 23:00:15.790498 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:15.790433    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:16.707849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:16.707725    5428 patch_prober.go:29] interesting pod/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 29 23:00:16.707849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:16.707779    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9" podUID="d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 29 23:00:16.791191 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:16.791154    5428 scope.go:122] "RemoveContainer" containerID="35bbeb63d44a72c5c5d2aea2d166e1eb673a150c957c4f68dbf5a5487fd8a303"
Oct 29 23:00:17.221601 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:17.221472    5428 generic.go:356] "Generic (PLEG): container finished" podID="080a2bc6-16d5-49ca-88c6-a41225b85c57" containerID="d9e2e990700c53319b99aa8ff4095f2817187194222f4dbed7509ae9d18be38b" exitCode=60
Oct 29 23:00:17.223215 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:17.221554    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4" event={"ID":"080a2bc6-16d5-49ca-88c6-a41225b85c57","Type":"ContainerDied","Data":"d9e2e990700c53319b99aa8ff4095f2817187194222f4dbed7509ae9d18be38b"}
Oct 29 23:00:17.223371 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:17.223255    5428 scope.go:122] "RemoveContainer" containerID="b4b950539f2ed132b89a4dbbce268c9edd618a01d9c92c5dfa638c9e52d46b48"
Oct 29 23:00:17.223830 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:17.223788    5428 scope.go:122] "RemoveContainer" containerID="d9e2e990700c53319b99aa8ff4095f2817187194222f4dbed7509ae9d18be38b"
Oct 29 23:00:17.224334 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:00:17.224308    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4_odh-ai-gateway-infra(080a2bc6-16d5-49ca-88c6-a41225b85c57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4" podUID="080a2bc6-16d5-49ca-88c6-a41225b85c57"
Oct 29 23:00:17.225357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:17.225331    5428 generic.go:356] "Generic (PLEG): container finished" podID="e63c04a5-4ccd-4e8c-8dee-ec131b926136" containerID="2987d79a3fb00f2657c9524db7b04e7515f467156eec4f961d302cc5994659c5" exitCode=60
Oct 29 23:00:17.225485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:17.225392    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw" event={"ID":"e63c04a5-4ccd-4e8c-8dee-ec131b926136","Type":"ContainerDied","Data":"2987d79a3fb00f2657c9524db7b04e7515f467156eec4f961d302cc5994659c5"}
Oct 29 23:00:17.225915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:17.225898    5428 scope.go:122] "RemoveContainer" containerID="2987d79a3fb00f2657c9524db7b04e7515f467156eec4f961d302cc5994659c5"
Oct 29 23:00:17.226302 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:00:17.226277    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw_odh-ai-gateway-infra(e63c04a5-4ccd-4e8c-8dee-ec131b926136)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw" podUID="e63c04a5-4ccd-4e8c-8dee-ec131b926136"
Oct 29 23:00:17.245582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:17.245561    5428 scope.go:122] "RemoveContainer" containerID="29d7538e34dcf564a621ec01f65e913c365032d341b97abfec679f83ad13cffc"
Oct 29 23:00:17.711824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:17.709406    5428 patch_prober.go:29] interesting pod/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 29 23:00:17.711824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:17.709482    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9" podUID="d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 29 23:00:17.755077 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:00:17.754906    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:00:17.780826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:17.780051    5428 scope.go:122] "RemoveContainer" containerID="b9e949f20f64d79cc880865572ffb107516a5694acb8ca393974b98d9e04bbf8"
Oct 29 23:00:18.091295 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.091216    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:18.327825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.326612    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4"]
Oct 29 23:00:18.338837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.338147    5428 generic.go:356] "Generic (PLEG): container finished" podID="e9d05b45-9d7e-424b-9313-c1551ceb45a3" containerID="206f4ea468de1d31a4ac3a3ea7bb79663164a34a52b9645fe77998716d98c31d" exitCode=60
Oct 29 23:00:18.338837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.338343    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz" event={"ID":"e9d05b45-9d7e-424b-9313-c1551ceb45a3","Type":"ContainerDied","Data":"206f4ea468de1d31a4ac3a3ea7bb79663164a34a52b9645fe77998716d98c31d"}
Oct 29 23:00:18.338837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.338397    5428 scope.go:122] "RemoveContainer" containerID="35bbeb63d44a72c5c5d2aea2d166e1eb673a150c957c4f68dbf5a5487fd8a303"
Oct 29 23:00:18.339687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.339236    5428 scope.go:122] "RemoveContainer" containerID="206f4ea468de1d31a4ac3a3ea7bb79663164a34a52b9645fe77998716d98c31d"
Oct 29 23:00:18.342821 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:00:18.341550    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz_odh-ai-gateway-infra(e9d05b45-9d7e-424b-9313-c1551ceb45a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz" podUID="e9d05b45-9d7e-424b-9313-c1551ceb45a3"
Oct 29 23:00:18.342821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.342455    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw"]
Oct 29 23:00:18.639457 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.639430    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4"
Oct 29 23:00:18.656571 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.656539    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080a2bc6-16d5-49ca-88c6-a41225b85c57-kube-api-access-wcx5t\" (UniqueName: \"kubernetes.io/projected/080a2bc6-16d5-49ca-88c6-a41225b85c57-kube-api-access-wcx5t\") pod \"080a2bc6-16d5-49ca-88c6-a41225b85c57\" (UID: \"080a2bc6-16d5-49ca-88c6-a41225b85c57\") "
Oct 29 23:00:18.656712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.656585    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/080a2bc6-16d5-49ca-88c6-a41225b85c57-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/080a2bc6-16d5-49ca-88c6-a41225b85c57-maas-api-service-ca\") pod \"080a2bc6-16d5-49ca-88c6-a41225b85c57\" (UID: \"080a2bc6-16d5-49ca-88c6-a41225b85c57\") "
Oct 29 23:00:18.656972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.656947    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080a2bc6-16d5-49ca-88c6-a41225b85c57-maas-api-service-ca" pod "080a2bc6-16d5-49ca-88c6-a41225b85c57" (UID: "080a2bc6-16d5-49ca-88c6-a41225b85c57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:18.657699 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.657672    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/080a2bc6-16d5-49ca-88c6-a41225b85c57-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:00:18.704079 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.703938    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080a2bc6-16d5-49ca-88c6-a41225b85c57-kube-api-access-wcx5t" pod "080a2bc6-16d5-49ca-88c6-a41225b85c57" (UID: "080a2bc6-16d5-49ca-88c6-a41225b85c57"). InnerVolumeSpecName "kube-api-access-wcx5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:18.708212 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.708179    5428 patch_prober.go:29] interesting pod/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 29 23:00:18.708331 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.708244    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9" podUID="d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 29 23:00:18.735020 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.734993    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw"
Oct 29 23:00:18.770850 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.758885    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e63c04a5-4ccd-4e8c-8dee-ec131b926136-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63c04a5-4ccd-4e8c-8dee-ec131b926136-maas-api-service-ca\") pod \"e63c04a5-4ccd-4e8c-8dee-ec131b926136\" (UID: \"e63c04a5-4ccd-4e8c-8dee-ec131b926136\") "
Oct 29 23:00:18.771364 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.771336    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63c04a5-4ccd-4e8c-8dee-ec131b926136-kube-api-access-9qxjz\" (UniqueName: \"kubernetes.io/projected/e63c04a5-4ccd-4e8c-8dee-ec131b926136-kube-api-access-9qxjz\") pod \"e63c04a5-4ccd-4e8c-8dee-ec131b926136\" (UID: \"e63c04a5-4ccd-4e8c-8dee-ec131b926136\") "
Oct 29 23:00:18.772829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.772783    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcx5t\" (UniqueName: \"kubernetes.io/projected/080a2bc6-16d5-49ca-88c6-a41225b85c57-kube-api-access-wcx5t\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:00:18.773414 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.759268    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63c04a5-4ccd-4e8c-8dee-ec131b926136-maas-api-service-ca" pod "e63c04a5-4ccd-4e8c-8dee-ec131b926136" (UID: "e63c04a5-4ccd-4e8c-8dee-ec131b926136"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:18.791569 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.791460    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63c04a5-4ccd-4e8c-8dee-ec131b926136-kube-api-access-9qxjz" pod "e63c04a5-4ccd-4e8c-8dee-ec131b926136" (UID: "e63c04a5-4ccd-4e8c-8dee-ec131b926136"). InnerVolumeSpecName "kube-api-access-9qxjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:18.890084 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.889898    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63c04a5-4ccd-4e8c-8dee-ec131b926136-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:00:18.890084 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:18.889928    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qxjz\" (UniqueName: \"kubernetes.io/projected/e63c04a5-4ccd-4e8c-8dee-ec131b926136-kube-api-access-9qxjz\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:00:19.393135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.393093    5428 generic.go:356] "Generic (PLEG): container finished" podID="8647546d-a3ef-427a-9942-2da7c19c3e08" containerID="96fb45382e5003e7baee8a4611c949f1f779bea266ba6991ea2c263208b57b42" exitCode=60
Oct 29 23:00:19.393313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.393162    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k" event={"ID":"8647546d-a3ef-427a-9942-2da7c19c3e08","Type":"ContainerDied","Data":"96fb45382e5003e7baee8a4611c949f1f779bea266ba6991ea2c263208b57b42"}
Oct 29 23:00:19.393313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.393245    5428 scope.go:122] "RemoveContainer" containerID="b9e949f20f64d79cc880865572ffb107516a5694acb8ca393974b98d9e04bbf8"
Oct 29 23:00:19.393923 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.393856    5428 scope.go:122] "RemoveContainer" containerID="96fb45382e5003e7baee8a4611c949f1f779bea266ba6991ea2c263208b57b42"
Oct 29 23:00:19.394941 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:00:19.394586    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k_odh-ai-gateway-infra(8647546d-a3ef-427a-9942-2da7c19c3e08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k" podUID="8647546d-a3ef-427a-9942-2da7c19c3e08"
Oct 29 23:00:19.396444 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.396416    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz"]
Oct 29 23:00:19.398676 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.398654    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4"
Oct 29 23:00:19.399252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.398652    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4" event={"ID":"080a2bc6-16d5-49ca-88c6-a41225b85c57","Type":"ContainerDied","Data":"48ee26afeed41d0f8d35b5534bde5e68d2a48ebf216f03aa917a070c9c817d69"}
Oct 29 23:00:19.400871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.400847    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw" event={"ID":"e63c04a5-4ccd-4e8c-8dee-ec131b926136","Type":"ContainerDied","Data":"02e96d8eb85418df39cf6c786f56280b42ef9761c1f4ecff650082b1446b4d0b"}
Oct 29 23:00:19.400993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.400895    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw"
Oct 29 23:00:19.414054 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.414028    5428 scope.go:122] "RemoveContainer" containerID="d9e2e990700c53319b99aa8ff4095f2817187194222f4dbed7509ae9d18be38b"
Oct 29 23:00:19.427496 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.427453    5428 scope.go:122] "RemoveContainer" containerID="2987d79a3fb00f2657c9524db7b04e7515f467156eec4f961d302cc5994659c5"
Oct 29 23:00:19.492732 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.492695    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4"]
Oct 29 23:00:19.498492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.498465    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-660540-29888580-f4bx4"]
Oct 29 23:00:19.545227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.545189    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz" podUID="3982f117-51f0-44b5-aa87-a5a6f3132fab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:19.596664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.596629    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw"]
Oct 29 23:00:19.601939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.601912    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888580-qw9tw"]
Oct 29 23:00:19.639665 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.639639    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz"
Oct 29 23:00:19.707769 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.707675    5428 patch_prober.go:29] interesting pod/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 29 23:00:19.707968 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.707760    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9" podUID="d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 29 23:00:19.707968 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.707918    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d05b45-9d7e-424b-9313-c1551ceb45a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d05b45-9d7e-424b-9313-c1551ceb45a3-maas-api-service-ca\") pod \"e9d05b45-9d7e-424b-9313-c1551ceb45a3\" (UID: \"e9d05b45-9d7e-424b-9313-c1551ceb45a3\") "
Oct 29 23:00:19.708094 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.708027    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d05b45-9d7e-424b-9313-c1551ceb45a3-kube-api-access-hmcfb\" (UniqueName: \"kubernetes.io/projected/e9d05b45-9d7e-424b-9313-c1551ceb45a3-kube-api-access-hmcfb\") pod \"e9d05b45-9d7e-424b-9313-c1551ceb45a3\" (UID: \"e9d05b45-9d7e-424b-9313-c1551ceb45a3\") "
Oct 29 23:00:19.708324 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.708300    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d05b45-9d7e-424b-9313-c1551ceb45a3-maas-api-service-ca" pod "e9d05b45-9d7e-424b-9313-c1551ceb45a3" (UID: "e9d05b45-9d7e-424b-9313-c1551ceb45a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:19.708942 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.708917    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d05b45-9d7e-424b-9313-c1551ceb45a3-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:00:19.710291 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.710265    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d05b45-9d7e-424b-9313-c1551ceb45a3-kube-api-access-hmcfb" pod "e9d05b45-9d7e-424b-9313-c1551ceb45a3" (UID: "e9d05b45-9d7e-424b-9313-c1551ceb45a3"). InnerVolumeSpecName "kube-api-access-hmcfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:19.811172 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:19.811134    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmcfb\" (UniqueName: \"kubernetes.io/projected/e9d05b45-9d7e-424b-9313-c1551ceb45a3-kube-api-access-hmcfb\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.004002 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.003905    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:20.135531 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.135425    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:20.413897 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.413861    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz" event={"ID":"e9d05b45-9d7e-424b-9313-c1551ceb45a3","Type":"ContainerDied","Data":"96c7927fcf2b9c7d8ac3a598b18c14ec43c54dda6de858d169d22f2145bce597"}
Oct 29 23:00:20.414093 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.413930    5428 scope.go:122] "RemoveContainer" containerID="206f4ea468de1d31a4ac3a3ea7bb79663164a34a52b9645fe77998716d98c31d"
Oct 29 23:00:20.414093 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.413878    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz"
Oct 29 23:00:20.450125 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.450089    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k"]
Oct 29 23:00:20.469151 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.469115    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz"]
Oct 29 23:00:20.496570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.496526    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c79a4e-29888580-zb9bz"]
Oct 29 23:00:20.660758 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.660724    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k"
Oct 29 23:00:20.712794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.712721    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 23:00:20.713856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.713834    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 23:00:20.736186 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.736152    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8647546d-a3ef-427a-9942-2da7c19c3e08-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8647546d-a3ef-427a-9942-2da7c19c3e08-maas-api-service-ca\") pod \"8647546d-a3ef-427a-9942-2da7c19c3e08\" (UID: \"8647546d-a3ef-427a-9942-2da7c19c3e08\") "
Oct 29 23:00:20.736425 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.736405    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8647546d-a3ef-427a-9942-2da7c19c3e08-kube-api-access-sqqqw\" (UniqueName: \"kubernetes.io/projected/8647546d-a3ef-427a-9942-2da7c19c3e08-kube-api-access-sqqqw\") pod \"8647546d-a3ef-427a-9942-2da7c19c3e08\" (UID: \"8647546d-a3ef-427a-9942-2da7c19c3e08\") "
Oct 29 23:00:20.736612 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.736585    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8647546d-a3ef-427a-9942-2da7c19c3e08-maas-api-service-ca" pod "8647546d-a3ef-427a-9942-2da7c19c3e08" (UID: "8647546d-a3ef-427a-9942-2da7c19c3e08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:20.737283 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.737258    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8647546d-a3ef-427a-9942-2da7c19c3e08-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.739479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.739444    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8647546d-a3ef-427a-9942-2da7c19c3e08-kube-api-access-sqqqw" pod "8647546d-a3ef-427a-9942-2da7c19c3e08" (UID: "8647546d-a3ef-427a-9942-2da7c19c3e08"). InnerVolumeSpecName "kube-api-access-sqqqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:20.785273 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.785237    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="080a2bc6-16d5-49ca-88c6-a41225b85c57" path="/var/lib/kubelet/pods/080a2bc6-16d5-49ca-88c6-a41225b85c57/volumes"
Oct 29 23:00:20.786061 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.786023    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e63c04a5-4ccd-4e8c-8dee-ec131b926136" path="/var/lib/kubelet/pods/e63c04a5-4ccd-4e8c-8dee-ec131b926136/volumes"
Oct 29 23:00:20.786879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.786677    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d05b45-9d7e-424b-9313-c1551ceb45a3" path="/var/lib/kubelet/pods/e9d05b45-9d7e-424b-9313-c1551ceb45a3/volumes"
Oct 29 23:00:20.839710 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:20.839667    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqqqw\" (UniqueName: \"kubernetes.io/projected/8647546d-a3ef-427a-9942-2da7c19c3e08-kube-api-access-sqqqw\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:00:21.373549 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:21.372722    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr"]
Oct 29 23:00:21.440895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:21.436957    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k"
Oct 29 23:00:21.440895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:21.437482    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k" event={"ID":"8647546d-a3ef-427a-9942-2da7c19c3e08","Type":"ContainerDied","Data":"f54578205cf40e61f69fc56ecaed7c57a48631716dda5b6bc2ab9e389bc2d37d"}
Oct 29 23:00:21.440895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:21.437535    5428 scope.go:122] "RemoveContainer" containerID="96fb45382e5003e7baee8a4611c949f1f779bea266ba6991ea2c263208b57b42"
Oct 29 23:00:21.499691 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:21.499609    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:21.509683 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:21.509641    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k"]
Oct 29 23:00:21.515306 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:21.515279    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-79e6c631-29888580-xns4k"]
Oct 29 23:00:21.619768 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:21.619732    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:21.734461 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:21.734368    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:22.604160 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:22.604115    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:22.781518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:22.781487    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8647546d-a3ef-427a-9942-2da7c19c3e08" path="/var/lib/kubelet/pods/8647546d-a3ef-427a-9942-2da7c19c3e08/volumes"
Oct 29 23:00:23.457710 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:23.457684    5428 generic.go:356] "Generic (PLEG): container finished" podID="3982f117-51f0-44b5-aa87-a5a6f3132fab" containerID="fb497befbae7bcbfc29c081478d54251e1d8a2433f1deb9acf602901a3ffcf3f" exitCode=0
Oct 29 23:00:23.458250 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:23.457760    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz" event={"ID":"3982f117-51f0-44b5-aa87-a5a6f3132fab","Type":"ContainerDied","Data":"fb497befbae7bcbfc29c081478d54251e1d8a2433f1deb9acf602901a3ffcf3f"}
Oct 29 23:00:23.631282 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:23.631251    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz"
Oct 29 23:00:23.732961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:23.732869    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3982f117-51f0-44b5-aa87-a5a6f3132fab-kube-api-access-gbhzp\" (UniqueName: \"kubernetes.io/projected/3982f117-51f0-44b5-aa87-a5a6f3132fab-kube-api-access-gbhzp\") pod \"3982f117-51f0-44b5-aa87-a5a6f3132fab\" (UID: \"3982f117-51f0-44b5-aa87-a5a6f3132fab\") "
Oct 29 23:00:23.733129 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:23.733028    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3982f117-51f0-44b5-aa87-a5a6f3132fab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3982f117-51f0-44b5-aa87-a5a6f3132fab-plugins-config-volume\") pod \"3982f117-51f0-44b5-aa87-a5a6f3132fab\" (UID: \"3982f117-51f0-44b5-aa87-a5a6f3132fab\") "
Oct 29 23:00:23.733483 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:23.733449    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3982f117-51f0-44b5-aa87-a5a6f3132fab-plugins-config-volume" pod "3982f117-51f0-44b5-aa87-a5a6f3132fab" (UID: "3982f117-51f0-44b5-aa87-a5a6f3132fab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:23.734234 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:23.734212    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3982f117-51f0-44b5-aa87-a5a6f3132fab-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:00:23.737015 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:23.736983    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3982f117-51f0-44b5-aa87-a5a6f3132fab-kube-api-access-gbhzp" pod "3982f117-51f0-44b5-aa87-a5a6f3132fab" (UID: "3982f117-51f0-44b5-aa87-a5a6f3132fab"). InnerVolumeSpecName "kube-api-access-gbhzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:23.836342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:23.836308    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbhzp\" (UniqueName: \"kubernetes.io/projected/3982f117-51f0-44b5-aa87-a5a6f3132fab-kube-api-access-gbhzp\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.313064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:24.313022    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:24.467348 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:24.467316    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz" event={"ID":"3982f117-51f0-44b5-aa87-a5a6f3132fab","Type":"ContainerDied","Data":"434c34d76988b6f2ec79ded82ac41f25ccf7847cc4b028d459650aa378ede234"}
Oct 29 23:00:24.467348 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:24.467336    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz"
Oct 29 23:00:24.467615 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:24.467419    5428 scope.go:122] "RemoveContainer" containerID="fb497befbae7bcbfc29c081478d54251e1d8a2433f1deb9acf602901a3ffcf3f"
Oct 29 23:00:24.518869 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:24.518830    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz"]
Oct 29 23:00:24.527931 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:24.527906    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79e6c631-7447f78679-2wfrz"]
Oct 29 23:00:24.703497 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:24.703455    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:24.778939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:24.778893    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3982f117-51f0-44b5-aa87-a5a6f3132fab" path="/var/lib/kubelet/pods/3982f117-51f0-44b5-aa87-a5a6f3132fab/volumes"
Oct 29 23:00:25.761908 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:25.761870    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:25.904224 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:25.904190    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:26.375098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:26.375061    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr" podUID="c47ec09e-792a-4344-af69-0d85b3f64125" containerName="payload-pre-processing" containerID="cri-o://c80493c215328b75783bd1610ec632248002fc3f04d825f75db0ee8ff6cd0e76" gracePeriod=25
Oct 29 23:00:26.679270 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:26.679244    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr"
Oct 29 23:00:26.799086 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:26.799059    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c47ec09e-792a-4344-af69-0d85b3f64125-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c47ec09e-792a-4344-af69-0d85b3f64125-plugins-config-volume\") pod \"c47ec09e-792a-4344-af69-0d85b3f64125\" (UID: \"c47ec09e-792a-4344-af69-0d85b3f64125\") "
Oct 29 23:00:26.799546 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:26.799112    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c47ec09e-792a-4344-af69-0d85b3f64125-kube-api-access-4dv4d\" (UniqueName: \"kubernetes.io/projected/c47ec09e-792a-4344-af69-0d85b3f64125-kube-api-access-4dv4d\") pod \"c47ec09e-792a-4344-af69-0d85b3f64125\" (UID: \"c47ec09e-792a-4344-af69-0d85b3f64125\") "
Oct 29 23:00:26.799546 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:26.799461    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c47ec09e-792a-4344-af69-0d85b3f64125-plugins-config-volume" pod "c47ec09e-792a-4344-af69-0d85b3f64125" (UID: "c47ec09e-792a-4344-af69-0d85b3f64125"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:26.800068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:26.800044    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c47ec09e-792a-4344-af69-0d85b3f64125-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:00:26.801635 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:26.801602    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c47ec09e-792a-4344-af69-0d85b3f64125-kube-api-access-4dv4d" pod "c47ec09e-792a-4344-af69-0d85b3f64125" (UID: "c47ec09e-792a-4344-af69-0d85b3f64125"). InnerVolumeSpecName "kube-api-access-4dv4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:26.902230 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:26.902157    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dv4d\" (UniqueName: \"kubernetes.io/projected/c47ec09e-792a-4344-af69-0d85b3f64125-kube-api-access-4dv4d\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:00:27.497223 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:27.497182    5428 generic.go:356] "Generic (PLEG): container finished" podID="c47ec09e-792a-4344-af69-0d85b3f64125" containerID="c80493c215328b75783bd1610ec632248002fc3f04d825f75db0ee8ff6cd0e76" exitCode=0
Oct 29 23:00:27.497422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:27.497282    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr"
Oct 29 23:00:27.497422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:27.497313    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr" event={"ID":"c47ec09e-792a-4344-af69-0d85b3f64125","Type":"ContainerDied","Data":"c80493c215328b75783bd1610ec632248002fc3f04d825f75db0ee8ff6cd0e76"}
Oct 29 23:00:27.497422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:27.497386    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr" event={"ID":"c47ec09e-792a-4344-af69-0d85b3f64125","Type":"ContainerDied","Data":"5ce57b8fd634c2be94147d4cf7bd1a5723ad65c43a24d66ec0338d04ae039b5c"}
Oct 29 23:00:27.497422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:27.497416    5428 scope.go:122] "RemoveContainer" containerID="c80493c215328b75783bd1610ec632248002fc3f04d825f75db0ee8ff6cd0e76"
Oct 29 23:00:27.506554 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:27.506528    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:27.513541 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:27.513516    5428 scope.go:122] "RemoveContainer" containerID="c80493c215328b75783bd1610ec632248002fc3f04d825f75db0ee8ff6cd0e76"
Oct 29 23:00:27.513914 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:00:27.513882    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80493c215328b75783bd1610ec632248002fc3f04d825f75db0ee8ff6cd0e76\": container with ID starting with c80493c215328b75783bd1610ec632248002fc3f04d825f75db0ee8ff6cd0e76 not found: ID does not exist" containerID="c80493c215328b75783bd1610ec632248002fc3f04d825f75db0ee8ff6cd0e76"
Oct 29 23:00:27.514038 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:27.513941    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80493c215328b75783bd1610ec632248002fc3f04d825f75db0ee8ff6cd0e76"} err="failed to get container status \"c80493c215328b75783bd1610ec632248002fc3f04d825f75db0ee8ff6cd0e76\": rpc error: code = NotFound desc = could not find container \"c80493c215328b75783bd1610ec632248002fc3f04d825f75db0ee8ff6cd0e76\": container with ID starting with c80493c215328b75783bd1610ec632248002fc3f04d825f75db0ee8ff6cd0e76 not found: ID does not exist"
Oct 29 23:00:27.553488 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:27.553440    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr"]
Oct 29 23:00:27.559153 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:27.559123    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-660540-5c5f794495-jfwpr"]
Oct 29 23:00:28.780395 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:28.780365    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c47ec09e-792a-4344-af69-0d85b3f64125" path="/var/lib/kubelet/pods/c47ec09e-792a-4344-af69-0d85b3f64125/volumes"
Oct 29 23:00:29.041919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:29.031372    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:29.660193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:29.660158    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:29.828528 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:29.828496    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:30.187552 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:30.187512    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:30.424301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:30.424269    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:31.997901 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:31.997864    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:33.431675 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:33.431645    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:34.048882 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:34.048837    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:34.316042 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:34.315709    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:34.752432 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:34.752394    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:34.864084 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:34.864049    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:35.254443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:35.254409    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:35.863150 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:35.863100    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:36.984665 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:36.984624    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:37.151812 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:37.151762    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:37.464196 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:37.464161    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:38.517573 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:38.517538    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:38.761973 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:38.761932    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:40.055821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:40.051263    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:40.206503 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:40.188778    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:40.411883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:40.349958    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:43.008534 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:43.004923    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:43.205265 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:43.205224    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-98p7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:00:43.234361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:43.234312    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b" podUID="bf52a0e8-c739-47f8-9676-c5261e8d0f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:43.234603 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:43.217701    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:43.450827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:43.425686    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 29 23:00:43.500101 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:43.500044    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:43.641674 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:43.641636    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:43.850278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:43.840908    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:44.150826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:44.150540    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:44.253387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:44.236197    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:44.275055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:44.257244    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 23:00:44.501013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:44.500910    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:44.547975 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:44.501759    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 23:00:44.575843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:44.541353    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj" podUID="63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded"
Oct 29 23:00:44.575843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:44.541399    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:44.596311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:44.595485    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 23:00:44.765364 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:44.765265    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 23:00:45.086882 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:45.009643    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 23:00:45.141023 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:45.140982    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:45.256587 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:45.256518    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:45.458829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:45.451962    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 23:00:45.593842 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:45.592889    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:45.785910 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:45.785821    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:47.157314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:47.157259    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:47.299101 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:47.299062    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:47.463436 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:00:47.463309    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:00:48.628750 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:48.628714    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:48.771976 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:48.771940    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:50.044319 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:50.044275    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:51.607191 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:51.607151    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:51.728409 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:51.728376    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:52.474920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:52.474888    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:53.906362 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:53.906320    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:54.065460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:54.065430    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:54.705000 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:54.704950    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:00:54.769545 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:54.769484    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752" podUID="9eeb8c61-ed7d-420a-b256-e14c3a750d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:55.166962 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:55.166930    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:57.230450 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:57.230410    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:59.208716 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:00:59.208683    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:00.910713 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:00.910664    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:01.704012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:01.703974    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:02.419330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:02.419292    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:02.908979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:02.908944    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:03.073154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:03.072401    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:03.836826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:03.836779    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:05.349947 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:05.349908    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:07.601697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:07.601612    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:07.721827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:07.721782    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:07.930080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:07.930040    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:08.318144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:08.318042    5428 generic.go:356] "Generic (PLEG): container finished" podID="9eeb8c61-ed7d-420a-b256-e14c3a750d9b" containerID="6608e117ee2250f826cadcc1a42e0eb48748a583a001124ba960d7039eabf579" exitCode=0
Oct 29 23:01:08.318319 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:08.318133    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752" event={"ID":"9eeb8c61-ed7d-420a-b256-e14c3a750d9b","Type":"ContainerDied","Data":"6608e117ee2250f826cadcc1a42e0eb48748a583a001124ba960d7039eabf579"}
Oct 29 23:01:08.318745 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:08.318724    5428 scope.go:122] "RemoveContainer" containerID="6608e117ee2250f826cadcc1a42e0eb48748a583a001124ba960d7039eabf579"
Oct 29 23:01:08.330856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:08.329290    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:01:08.330856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:08.329402    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:01:08.330856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:08.329438    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:01:08.663728 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:08.663691    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 23:01:09.309729 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:09.309682    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:09.338320 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:09.338268    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752" event={"ID":"9eeb8c61-ed7d-420a-b256-e14c3a750d9b","Type":"ContainerStarted","Data":"48469a7175d1eada11ba0d3b73602ff66e162dc95523e5d467ecf837c315a0ba"}
Oct 29 23:01:09.448931 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:09.448889    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:10.023137 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.023102    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 23:01:10.024450 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.024418    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 29 23:01:10.025243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.024486    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752" podUID="9eeb8c61-ed7d-420a-b256-e14c3a750d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 29 23:01:10.349703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.349620    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"]
Oct 29 23:01:10.350528 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350502    5428 cpu_manager.go:414] "containerMap: removing container" podUID="080a2bc6-16d5-49ca-88c6-a41225b85c57" containerName="cleanup"
Oct 29 23:01:10.350528 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350527    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080a2bc6-16d5-49ca-88c6-a41225b85c57" containerName="cleanup"
Oct 29 23:01:10.350712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350538    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e63c04a5-4ccd-4e8c-8dee-ec131b926136" containerName="cleanup"
Oct 29 23:01:10.350712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350545    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63c04a5-4ccd-4e8c-8dee-ec131b926136" containerName="cleanup"
Oct 29 23:01:10.350712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350558    5428 cpu_manager.go:414] "containerMap: removing container" podUID="8647546d-a3ef-427a-9942-2da7c19c3e08" containerName="cleanup"
Oct 29 23:01:10.350712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350565    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8647546d-a3ef-427a-9942-2da7c19c3e08" containerName="cleanup"
Oct 29 23:01:10.350712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350585    5428 cpu_manager.go:414] "containerMap: removing container" podUID="c47ec09e-792a-4344-af69-0d85b3f64125" containerName="payload-pre-processing"
Oct 29 23:01:10.350712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350593    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c47ec09e-792a-4344-af69-0d85b3f64125" containerName="payload-pre-processing"
Oct 29 23:01:10.350712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350618    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e9d05b45-9d7e-424b-9313-c1551ceb45a3" containerName="cleanup"
Oct 29 23:01:10.350712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350624    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d05b45-9d7e-424b-9313-c1551ceb45a3" containerName="cleanup"
Oct 29 23:01:10.350712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350633    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e9d05b45-9d7e-424b-9313-c1551ceb45a3" containerName="cleanup"
Oct 29 23:01:10.350712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350639    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d05b45-9d7e-424b-9313-c1551ceb45a3" containerName="cleanup"
Oct 29 23:01:10.350712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350652    5428 cpu_manager.go:414] "containerMap: removing container" podUID="8647546d-a3ef-427a-9942-2da7c19c3e08" containerName="cleanup"
Oct 29 23:01:10.350712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350658    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8647546d-a3ef-427a-9942-2da7c19c3e08" containerName="cleanup"
Oct 29 23:01:10.350712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350677    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e9d05b45-9d7e-424b-9313-c1551ceb45a3" containerName="cleanup"
Oct 29 23:01:10.350712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350684    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d05b45-9d7e-424b-9313-c1551ceb45a3" containerName="cleanup"
Oct 29 23:01:10.350712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350710    5428 cpu_manager.go:414] "containerMap: removing container" podUID="3982f117-51f0-44b5-aa87-a5a6f3132fab" containerName="payload-pre-processing"
Oct 29 23:01:10.350712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350717    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3982f117-51f0-44b5-aa87-a5a6f3132fab" containerName="payload-pre-processing"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350743    5428 cpu_manager.go:414] "containerMap: removing container" podUID="080a2bc6-16d5-49ca-88c6-a41225b85c57" containerName="cleanup"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350749    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080a2bc6-16d5-49ca-88c6-a41225b85c57" containerName="cleanup"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350776    5428 cpu_manager.go:414] "containerMap: removing container" podUID="8647546d-a3ef-427a-9942-2da7c19c3e08" containerName="cleanup"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350784    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8647546d-a3ef-427a-9942-2da7c19c3e08" containerName="cleanup"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350796    5428 cpu_manager.go:414] "containerMap: removing container" podUID="1a434fd6-4f8e-4a9d-a7c8-389a718a45f8" containerName="revoke-keys"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350817    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a434fd6-4f8e-4a9d-a7c8-389a718a45f8" containerName="revoke-keys"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350836    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e63c04a5-4ccd-4e8c-8dee-ec131b926136" containerName="cleanup"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350842    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63c04a5-4ccd-4e8c-8dee-ec131b926136" containerName="cleanup"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350855    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e63c04a5-4ccd-4e8c-8dee-ec131b926136" containerName="cleanup"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.350859    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63c04a5-4ccd-4e8c-8dee-ec131b926136" containerName="cleanup"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.351000    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="080a2bc6-16d5-49ca-88c6-a41225b85c57" containerName="cleanup"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.351008    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63c04a5-4ccd-4e8c-8dee-ec131b926136" containerName="cleanup"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.351015    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8647546d-a3ef-427a-9942-2da7c19c3e08" containerName="cleanup"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.351029    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c47ec09e-792a-4344-af69-0d85b3f64125" containerName="payload-pre-processing"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.351038    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d05b45-9d7e-424b-9313-c1551ceb45a3" containerName="cleanup"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.351044    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d05b45-9d7e-424b-9313-c1551ceb45a3" containerName="cleanup"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.351050    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8647546d-a3ef-427a-9942-2da7c19c3e08" containerName="cleanup"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.351057    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d05b45-9d7e-424b-9313-c1551ceb45a3" containerName="cleanup"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.351064    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3982f117-51f0-44b5-aa87-a5a6f3132fab" containerName="payload-pre-processing"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.351069    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="080a2bc6-16d5-49ca-88c6-a41225b85c57" containerName="cleanup"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.351077    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a434fd6-4f8e-4a9d-a7c8-389a718a45f8" containerName="revoke-keys"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.351085    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63c04a5-4ccd-4e8c-8dee-ec131b926136" containerName="cleanup"
Oct 29 23:01:10.352693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.351091    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63c04a5-4ccd-4e8c-8dee-ec131b926136" containerName="cleanup"
Oct 29 23:01:10.355076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.354387    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"
Oct 29 23:01:10.445876 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.445833    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d90c5f-4afa-43f1-8574-4e271260f1c3-metrics-certs\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf\" (UID: \"71d90c5f-4afa-43f1-8574-4e271260f1c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"
Oct 29 23:01:10.446066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.445880    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7t62\" (UniqueName: \"kubernetes.io/projected/71d90c5f-4afa-43f1-8574-4e271260f1c3-kube-api-access-l7t62\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf\" (UID: \"71d90c5f-4afa-43f1-8574-4e271260f1c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"
Oct 29 23:01:10.446186 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.446163    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71d90c5f-4afa-43f1-8574-4e271260f1c3-openshift-service-ca\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf\" (UID: \"71d90c5f-4afa-43f1-8574-4e271260f1c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"
Oct 29 23:01:10.446330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.446313    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71d90c5f-4afa-43f1-8574-4e271260f1c3-maas-api-tls\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf\" (UID: \"71d90c5f-4afa-43f1-8574-4e271260f1c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"
Oct 29 23:01:10.481826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.472075    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-79e6c631-serving-cert\""
Oct 29 23:01:10.491003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.490582    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:10.498338 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.497595    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"]
Oct 29 23:01:10.548057 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.548022    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71d90c5f-4afa-43f1-8574-4e271260f1c3-openshift-service-ca\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf\" (UID: \"71d90c5f-4afa-43f1-8574-4e271260f1c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"
Oct 29 23:01:10.548223 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.548177    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71d90c5f-4afa-43f1-8574-4e271260f1c3-maas-api-tls\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf\" (UID: \"71d90c5f-4afa-43f1-8574-4e271260f1c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"
Oct 29 23:01:10.548813 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.548431    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d90c5f-4afa-43f1-8574-4e271260f1c3-metrics-certs\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf\" (UID: \"71d90c5f-4afa-43f1-8574-4e271260f1c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"
Oct 29 23:01:10.548813 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.548459    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7t62\" (UniqueName: \"kubernetes.io/projected/71d90c5f-4afa-43f1-8574-4e271260f1c3-kube-api-access-l7t62\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf\" (UID: \"71d90c5f-4afa-43f1-8574-4e271260f1c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"
Oct 29 23:01:10.549008 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.548991    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71d90c5f-4afa-43f1-8574-4e271260f1c3-openshift-service-ca\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf\" (UID: \"71d90c5f-4afa-43f1-8574-4e271260f1c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"
Oct 29 23:01:10.553195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.553098    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71d90c5f-4afa-43f1-8574-4e271260f1c3-maas-api-tls\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf\" (UID: \"71d90c5f-4afa-43f1-8574-4e271260f1c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"
Oct 29 23:01:10.553429 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.553398    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d90c5f-4afa-43f1-8574-4e271260f1c3-metrics-certs\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf\" (UID: \"71d90c5f-4afa-43f1-8574-4e271260f1c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"
Oct 29 23:01:10.605083 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.604940    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7t62\" (UniqueName: \"kubernetes.io/projected/71d90c5f-4afa-43f1-8574-4e271260f1c3-kube-api-access-l7t62\") pod \"maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf\" (UID: \"71d90c5f-4afa-43f1-8574-4e271260f1c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"
Oct 29 23:01:10.623914 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.623877    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:10.698216 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.698163    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"
Oct 29 23:01:10.744040 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:10.744006    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:11.022779 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:11.022747    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 29 23:01:11.022975 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:11.022813    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752" podUID="9eeb8c61-ed7d-420a-b256-e14c3a750d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 29 23:01:11.727430 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:11.727382    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"]
Oct 29 23:01:11.737320 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:01:11.737276    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d90c5f_4afa_43f1_8574_4e271260f1c3.slice/crio-f3164685e6664fbdcb4594477235c4ea354cf4b1fed19bc3ae276459eeb95667 WatchSource:0}: Error finding container f3164685e6664fbdcb4594477235c4ea354cf4b1fed19bc3ae276459eeb95667: Status 404 returned error can't find the container with id f3164685e6664fbdcb4594477235c4ea354cf4b1fed19bc3ae276459eeb95667
Oct 29 23:01:12.023532 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:12.023454    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 29 23:01:12.023679 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:12.023529    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752" podUID="9eeb8c61-ed7d-420a-b256-e14c3a750d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 29 23:01:12.404098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:12.404053    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf" event={"ID":"71d90c5f-4afa-43f1-8574-4e271260f1c3","Type":"ContainerStarted","Data":"6ccf3271cf2b0bef3e9f15a51ce3863aaccc93dc778d32d76d578bfad5ab63b1"}
Oct 29 23:01:12.407816 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:12.404109    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf" event={"ID":"71d90c5f-4afa-43f1-8574-4e271260f1c3","Type":"ContainerStarted","Data":"f3164685e6664fbdcb4594477235c4ea354cf4b1fed19bc3ae276459eeb95667"}
Oct 29 23:01:12.407816 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:12.405486    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"
Oct 29 23:01:12.447215 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:12.447161    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf" podStartSLOduration=1.965776263 podStartE2EDuration="2.447147005s" totalImagesPullingTime="481.370742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:12.443404134 +0000 UTC m=+1075.786352879" watchObservedRunningTime="2026-10-29 23:01:12.447147005 +0000 UTC m=+1075.790095751"
Oct 29 23:01:12.821272 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:12.821229    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:13.023003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:13.022971    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 29 23:01:13.023219 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:13.023025    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752" podUID="9eeb8c61-ed7d-420a-b256-e14c3a750d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 29 23:01:14.023204 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:14.023172    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 29 23:01:14.023616 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:14.023223    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752" podUID="9eeb8c61-ed7d-420a-b256-e14c3a750d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 29 23:01:14.197587 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:14.197548    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:14.688105 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:14.688056    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:15.041037 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:15.040946    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 23:01:15.041602 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:15.041579    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 23:01:15.044863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:15.043232    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 23:01:15.460523 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:15.460478    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:15.931383 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:15.931340    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:17.111225 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:17.111175    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:17.331815 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:17.331715    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:17.422867 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:01:17.422742    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:01:18.944080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:18.944046    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:19.144906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:19.144870    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:19.264600 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:19.264508    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:19.424899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:19.424872    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"
Oct 29 23:01:20.756540 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:20.756498    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:20.984248 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:20.984207    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:21.792990 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:21.792959    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:22.456341 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:22.456305    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:22.594887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:22.594853    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:24.197829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:24.197782    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:24.918095 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:24.918056    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:25.883729 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:25.883689    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:26.339227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:26.339125    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:28.029928 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:28.029793    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:28.269236 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:28.269207    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:30.056769 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:30.056730    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:31.325599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:31.325559    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:31.435048 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:31.435012    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:31.593866 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:31.593745    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:31.957093 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:31.957055    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:33.506827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:33.505868    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:35.708270 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:35.708234    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:35.945299 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:35.945257    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:36.264175 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:36.264137    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:37.001958 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:37.001925    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:37.905083 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:37.905047    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:38.547718 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:38.547673    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:38.677677 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:38.677641    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:38.877541 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:38.877494    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"]
Oct 29 23:01:38.878646 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:38.877850    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p" podUID="c1804496-00d1-4679-a069-89a1caec4e75" containerName="istio-proxy" containerID="cri-o://91e843f7914e7e8cbd4575f6fbedee8d396604fd3d062ee749133889a608a02d" gracePeriod=30
Oct 29 23:01:38.923117 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:38.923084    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:39.358599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:39.358514    5428 patch_prober.go:29] interesting pod/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:01:39.358599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:39.358574    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p" podUID="c1804496-00d1-4679-a069-89a1caec4e75" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:01:39.890394 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:39.890358    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:40.095565 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:40.095533    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:40.442569 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:40.442526    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:40.557751 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:40.557688    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:41.259082 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:41.259040    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:41.372080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:41.372045    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:41.507775 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:41.507686    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:43.151706 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.151670    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"]
Oct 29 23:01:43.153752 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.153729    5428 cpu_manager.go:414] "containerMap: removing container" podUID="080a2bc6-16d5-49ca-88c6-a41225b85c57" containerName="cleanup"
Oct 29 23:01:43.154682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.154668    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080a2bc6-16d5-49ca-88c6-a41225b85c57" containerName="cleanup"
Oct 29 23:01:43.155342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.155330    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8647546d-a3ef-427a-9942-2da7c19c3e08" containerName="cleanup"
Oct 29 23:01:43.155381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.155348    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="080a2bc6-16d5-49ca-88c6-a41225b85c57" containerName="cleanup"
Oct 29 23:01:43.158273 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.158253    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"
Oct 29 23:01:43.179008 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.178981    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"]
Oct 29 23:01:43.182656 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.182631    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-7c486c-serving-cert\""
Oct 29 23:01:43.255591 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.255546    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56f0cf7-d334-4139-b135-8558427a0095-openshift-service-ca\") pod \"maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr\" (UID: \"c56f0cf7-d334-4139-b135-8558427a0095\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"
Oct 29 23:01:43.255864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.255736    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56f0cf7-d334-4139-b135-8558427a0095-maas-api-tls\") pod \"maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr\" (UID: \"c56f0cf7-d334-4139-b135-8558427a0095\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"
Oct 29 23:01:43.255974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.255886    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56f0cf7-d334-4139-b135-8558427a0095-metrics-certs\") pod \"maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr\" (UID: \"c56f0cf7-d334-4139-b135-8558427a0095\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"
Oct 29 23:01:43.256078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.256051    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsmlf\" (UniqueName: \"kubernetes.io/projected/c56f0cf7-d334-4139-b135-8558427a0095-kube-api-access-dsmlf\") pod \"maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr\" (UID: \"c56f0cf7-d334-4139-b135-8558427a0095\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"
Oct 29 23:01:43.259355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.259329    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:43.358201 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.358169    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56f0cf7-d334-4139-b135-8558427a0095-openshift-service-ca\") pod \"maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr\" (UID: \"c56f0cf7-d334-4139-b135-8558427a0095\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"
Oct 29 23:01:43.358376 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.358233    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56f0cf7-d334-4139-b135-8558427a0095-maas-api-tls\") pod \"maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr\" (UID: \"c56f0cf7-d334-4139-b135-8558427a0095\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"
Oct 29 23:01:43.358376 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.358284    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56f0cf7-d334-4139-b135-8558427a0095-metrics-certs\") pod \"maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr\" (UID: \"c56f0cf7-d334-4139-b135-8558427a0095\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"
Oct 29 23:01:43.359300 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.358375    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsmlf\" (UniqueName: \"kubernetes.io/projected/c56f0cf7-d334-4139-b135-8558427a0095-kube-api-access-dsmlf\") pod \"maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr\" (UID: \"c56f0cf7-d334-4139-b135-8558427a0095\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"
Oct 29 23:01:43.359300 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.358925    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56f0cf7-d334-4139-b135-8558427a0095-openshift-service-ca\") pod \"maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr\" (UID: \"c56f0cf7-d334-4139-b135-8558427a0095\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"
Oct 29 23:01:43.368483 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.368457    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56f0cf7-d334-4139-b135-8558427a0095-maas-api-tls\") pod \"maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr\" (UID: \"c56f0cf7-d334-4139-b135-8558427a0095\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"
Oct 29 23:01:43.368575 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.368467    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56f0cf7-d334-4139-b135-8558427a0095-metrics-certs\") pod \"maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr\" (UID: \"c56f0cf7-d334-4139-b135-8558427a0095\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"
Oct 29 23:01:43.371570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.371544    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:43.382017 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.381993    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsmlf\" (UniqueName: \"kubernetes.io/projected/c56f0cf7-d334-4139-b135-8558427a0095-kube-api-access-dsmlf\") pod \"maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr\" (UID: \"c56f0cf7-d334-4139-b135-8558427a0095\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"
Oct 29 23:01:43.487957 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.487858    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:43.500837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.500792    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"
Oct 29 23:01:43.965792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.965757    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:43.967666 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:43.967634    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"]
Oct 29 23:01:43.987058 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:01:43.987028    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56f0cf7_d334_4139_b135_8558427a0095.slice/crio-374509db29a35946ad4f4b7cd127d923edb981647591161626da674d3f17d109 WatchSource:0}: Error finding container 374509db29a35946ad4f4b7cd127d923edb981647591161626da674d3f17d109: Status 404 returned error can't find the container with id 374509db29a35946ad4f4b7cd127d923edb981647591161626da674d3f17d109
Oct 29 23:01:44.182962 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.182901    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 23:01:44.267942 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.267886    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:44.277792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.277755    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1804496-00d1-4679-a069-89a1caec4e75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1804496-00d1-4679-a069-89a1caec4e75-istiod-ca-cert\") pod \"c1804496-00d1-4679-a069-89a1caec4e75\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") "
Oct 29 23:01:44.278542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.277972    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1804496-00d1-4679-a069-89a1caec4e75-kube-api-access-mxv47\" (UniqueName: \"kubernetes.io/projected/c1804496-00d1-4679-a069-89a1caec4e75-kube-api-access-mxv47\") pod \"c1804496-00d1-4679-a069-89a1caec4e75\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") "
Oct 29 23:01:44.278542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.278020    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1804496-00d1-4679-a069-89a1caec4e75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1804496-00d1-4679-a069-89a1caec4e75-istio-podinfo\") pod \"c1804496-00d1-4679-a069-89a1caec4e75\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") "
Oct 29 23:01:44.278542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.278138    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-credential-socket\") pod \"c1804496-00d1-4679-a069-89a1caec4e75\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") "
Oct 29 23:01:44.278542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.278165    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-workload-certs\") pod \"c1804496-00d1-4679-a069-89a1caec4e75\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") "
Oct 29 23:01:44.278542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.278379    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1804496-00d1-4679-a069-89a1caec4e75-istio-token\" (UniqueName: \"kubernetes.io/projected/c1804496-00d1-4679-a069-89a1caec4e75-istio-token\") pod \"c1804496-00d1-4679-a069-89a1caec4e75\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") "
Oct 29 23:01:44.278542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.278464    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-workload-socket\") pod \"c1804496-00d1-4679-a069-89a1caec4e75\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") "
Oct 29 23:01:44.278542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.278504    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-istio-data\") pod \"c1804496-00d1-4679-a069-89a1caec4e75\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") "
Oct 29 23:01:44.282487 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.278669    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-istio-envoy\") pod \"c1804496-00d1-4679-a069-89a1caec4e75\" (UID: \"c1804496-00d1-4679-a069-89a1caec4e75\") "
Oct 29 23:01:44.282487 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.281089    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-credential-socket" pod "c1804496-00d1-4679-a069-89a1caec4e75" (UID: "c1804496-00d1-4679-a069-89a1caec4e75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:44.282487 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.281316    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-workload-socket" pod "c1804496-00d1-4679-a069-89a1caec4e75" (UID: "c1804496-00d1-4679-a069-89a1caec4e75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:44.282487 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.281630    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-istio-data" pod "c1804496-00d1-4679-a069-89a1caec4e75" (UID: "c1804496-00d1-4679-a069-89a1caec4e75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:44.283721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.283650    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-workload-certs" pod "c1804496-00d1-4679-a069-89a1caec4e75" (UID: "c1804496-00d1-4679-a069-89a1caec4e75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:44.284748 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.284725    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"]
Oct 29 23:01:44.286096 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.286074    5428 cpu_manager.go:414] "containerMap: removing container" podUID="c1804496-00d1-4679-a069-89a1caec4e75" containerName="istio-proxy"
Oct 29 23:01:44.286233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.286217    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1804496-00d1-4679-a069-89a1caec4e75" containerName="istio-proxy"
Oct 29 23:01:44.286372 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.286358    5428 cpu_manager.go:414] "containerMap: removing container" podUID="c1804496-00d1-4679-a069-89a1caec4e75" containerName="istio-proxy"
Oct 29 23:01:44.286468 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.286453    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1804496-00d1-4679-a069-89a1caec4e75" containerName="istio-proxy"
Oct 29 23:01:44.286887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.286868    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1804496-00d1-4679-a069-89a1caec4e75" containerName="istio-proxy"
Oct 29 23:01:44.289791 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.289712    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1804496-00d1-4679-a069-89a1caec4e75-istiod-ca-cert" pod "c1804496-00d1-4679-a069-89a1caec4e75" (UID: "c1804496-00d1-4679-a069-89a1caec4e75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:44.291062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.291033    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.292209 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.292187    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-istio-envoy" pod "c1804496-00d1-4679-a069-89a1caec4e75" (UID: "c1804496-00d1-4679-a069-89a1caec4e75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:44.292668 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.292639    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1804496-00d1-4679-a069-89a1caec4e75-istio-token" pod "c1804496-00d1-4679-a069-89a1caec4e75" (UID: "c1804496-00d1-4679-a069-89a1caec4e75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:44.293238 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.293215    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1804496-00d1-4679-a069-89a1caec4e75-kube-api-access-mxv47" pod "c1804496-00d1-4679-a069-89a1caec4e75" (UID: "c1804496-00d1-4679-a069-89a1caec4e75"). InnerVolumeSpecName "kube-api-access-mxv47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:44.293851 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.293549    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1804496-00d1-4679-a069-89a1caec4e75-istio-podinfo" pod "c1804496-00d1-4679-a069-89a1caec4e75" (UID: "c1804496-00d1-4679-a069-89a1caec4e75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:44.322081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.322035    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"]
Oct 29 23:01:44.428824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.385015    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.428824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.385124    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.428824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.385161    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4118f7d-454f-4162-8bb5-f58214106d7d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.428824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.385287    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.428824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.385345    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w5tk\" (UniqueName: \"kubernetes.io/projected/d4118f7d-454f-4162-8bb5-f58214106d7d-kube-api-access-2w5tk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.428824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.385408    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.428824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.385473    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.428824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.385576    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:01:44.428824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.385593    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1804496-00d1-4679-a069-89a1caec4e75-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:01:44.428824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.385606    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxv47\" (UniqueName: \"kubernetes.io/projected/c1804496-00d1-4679-a069-89a1caec4e75-kube-api-access-mxv47\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:01:44.428824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.385622    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1804496-00d1-4679-a069-89a1caec4e75-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:01:44.428824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.385636    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:01:44.428824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.385651    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:01:44.428824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.385665    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1804496-00d1-4679-a069-89a1caec4e75-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:01:44.428824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.385680    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:01:44.428824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.385692    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1804496-00d1-4679-a069-89a1caec4e75-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:01:44.533044 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.489649    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.533044 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.489759    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.545157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.533983    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.600024 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.489796    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4118f7d-454f-4162-8bb5-f58214106d7d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.600553 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.600447    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.602203 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.602178    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w5tk\" (UniqueName: \"kubernetes.io/projected/d4118f7d-454f-4162-8bb5-f58214106d7d-kube-api-access-2w5tk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.602385 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.602368    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.602525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.602513    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.619748 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.618563    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4118f7d-454f-4162-8bb5-f58214106d7d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.620305 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.620248    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.620843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.620823    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.648366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.647995    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.651505 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.651430    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w5tk\" (UniqueName: \"kubernetes.io/projected/d4118f7d-454f-4162-8bb5-f58214106d7d-kube-api-access-2w5tk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.670563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.661306    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmvx7\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.841624 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.841593    5428 generic.go:356] "Generic (PLEG): container finished" podID="c1804496-00d1-4679-a069-89a1caec4e75" containerID="91e843f7914e7e8cbd4575f6fbedee8d396604fd3d062ee749133889a608a02d" exitCode=0
Oct 29 23:01:44.841745 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.841642    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"
Oct 29 23:01:44.841745 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.841653    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p" event={"ID":"c1804496-00d1-4679-a069-89a1caec4e75","Type":"ContainerDied","Data":"91e843f7914e7e8cbd4575f6fbedee8d396604fd3d062ee749133889a608a02d"}
Oct 29 23:01:44.841745 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.841715    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p" event={"ID":"c1804496-00d1-4679-a069-89a1caec4e75","Type":"ContainerDied","Data":"b7a137c30a0b66c8bb496a69b1937d5b3b0ddf97ce546e4b3bdbc20535dfc6d0"}
Oct 29 23:01:44.843128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.841759    5428 scope.go:122] "RemoveContainer" containerID="91e843f7914e7e8cbd4575f6fbedee8d396604fd3d062ee749133889a608a02d"
Oct 29 23:01:44.844312 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.844285    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr" event={"ID":"c56f0cf7-d334-4139-b135-8558427a0095","Type":"ContainerStarted","Data":"f2a76a2dabd0c5eeb8c4f5f1a8a41f9c2d3962bee174f7415c3240e2a855adf6"}
Oct 29 23:01:44.844434 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.844336    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr" event={"ID":"c56f0cf7-d334-4139-b135-8558427a0095","Type":"ContainerStarted","Data":"374509db29a35946ad4f4b7cd127d923edb981647591161626da674d3f17d109"}
Oct 29 23:01:44.844490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.844448    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"
Oct 29 23:01:44.856262 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.856233    5428 scope.go:122] "RemoveContainer" containerID="24d1bd4a76f86cd41d477586472dd4f80b80aced81c3182a213ec406bf54a425"
Oct 29 23:01:44.868750 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.868730    5428 scope.go:122] "RemoveContainer" containerID="91e843f7914e7e8cbd4575f6fbedee8d396604fd3d062ee749133889a608a02d"
Oct 29 23:01:44.869877 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:01:44.869754    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e843f7914e7e8cbd4575f6fbedee8d396604fd3d062ee749133889a608a02d\": container with ID starting with 91e843f7914e7e8cbd4575f6fbedee8d396604fd3d062ee749133889a608a02d not found: ID does not exist" containerID="91e843f7914e7e8cbd4575f6fbedee8d396604fd3d062ee749133889a608a02d"
Oct 29 23:01:44.869877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.869850    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e843f7914e7e8cbd4575f6fbedee8d396604fd3d062ee749133889a608a02d"} err="failed to get container status \"91e843f7914e7e8cbd4575f6fbedee8d396604fd3d062ee749133889a608a02d\": rpc error: code = NotFound desc = could not find container \"91e843f7914e7e8cbd4575f6fbedee8d396604fd3d062ee749133889a608a02d\": container with ID starting with 91e843f7914e7e8cbd4575f6fbedee8d396604fd3d062ee749133889a608a02d not found: ID does not exist"
Oct 29 23:01:44.869877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.869875    5428 scope.go:122] "RemoveContainer" containerID="24d1bd4a76f86cd41d477586472dd4f80b80aced81c3182a213ec406bf54a425"
Oct 29 23:01:44.870536 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:01:44.870210    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d1bd4a76f86cd41d477586472dd4f80b80aced81c3182a213ec406bf54a425\": container with ID starting with 24d1bd4a76f86cd41d477586472dd4f80b80aced81c3182a213ec406bf54a425 not found: ID does not exist" containerID="24d1bd4a76f86cd41d477586472dd4f80b80aced81c3182a213ec406bf54a425"
Oct 29 23:01:44.870536 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.870243    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d1bd4a76f86cd41d477586472dd4f80b80aced81c3182a213ec406bf54a425"} err="failed to get container status \"24d1bd4a76f86cd41d477586472dd4f80b80aced81c3182a213ec406bf54a425\": rpc error: code = NotFound desc = could not find container \"24d1bd4a76f86cd41d477586472dd4f80b80aced81c3182a213ec406bf54a425\": container with ID starting with 24d1bd4a76f86cd41d477586472dd4f80b80aced81c3182a213ec406bf54a425 not found: ID does not exist"
Oct 29 23:01:44.902333 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.902277    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr" podStartSLOduration=1.192689679 podStartE2EDuration="1.902261476s" totalImagesPullingTime="709.571797ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:44.896193241 +0000 UTC m=+1108.239141987" watchObservedRunningTime="2026-10-29 23:01:44.902261476 +0000 UTC m=+1108.245210221"
Oct 29 23:01:44.905904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.905876    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:44.940587 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.940553    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"]
Oct 29 23:01:44.945290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:44.945261    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8b57a2-openshift-default-dc9567d5c-jgf2p"]
Oct 29 23:01:45.258453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:45.258422    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"]
Oct 29 23:01:45.268326 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:45.268297    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:45.272146 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:01:45.272116    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4118f7d_454f_4162_8bb5_f58214106d7d.slice/crio-42335d7d0649eaa881569daf581c985a8556258fd52e1ca17ad81d76d1e2c7fb WatchSource:0}: Error finding container 42335d7d0649eaa881569daf581c985a8556258fd52e1ca17ad81d76d1e2c7fb: Status 404 returned error can't find the container with id 42335d7d0649eaa881569daf581c985a8556258fd52e1ca17ad81d76d1e2c7fb
Oct 29 23:01:45.276673 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:45.276637    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:01:45.276947 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:45.276728    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:01:45.406979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:45.406932    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:45.854193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:45.854156    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7" event={"ID":"d4118f7d-454f-4162-8bb5-f58214106d7d","Type":"ContainerStarted","Data":"29004051e9ce3d93ba8a32a9824171f0195861091decd27be2504eb4faf4e928"}
Oct 29 23:01:45.854403 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:45.854217    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7" event={"ID":"d4118f7d-454f-4162-8bb5-f58214106d7d","Type":"ContainerStarted","Data":"42335d7d0649eaa881569daf581c985a8556258fd52e1ca17ad81d76d1e2c7fb"}
Oct 29 23:01:45.854403 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:45.854245    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:45.923089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:45.923014    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7" podStartSLOduration=1.922994048 podStartE2EDuration="1.922994048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:45.918681886 +0000 UTC m=+1109.261630632" watchObservedRunningTime="2026-10-29 23:01:45.922994048 +0000 UTC m=+1109.265942795"
Oct 29 23:01:46.091421 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:46.091391    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:46.782828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:46.782775    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1804496-00d1-4679-a069-89a1caec4e75" path="/var/lib/kubelet/pods/c1804496-00d1-4679-a069-89a1caec4e75/volumes"
Oct 29 23:01:46.868314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:46.868273    5428 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-jmvx7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:01:46.868501 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:46.868338    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7" podUID="d4118f7d-454f-4162-8bb5-f58214106d7d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:01:46.956117 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:46.956082    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:47.088921 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:47.088837    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:47.407510 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:01:47.407465    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:01:48.469952 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:48.469904    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:49.867436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:49.867403    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:01:50.842995 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:50.842957    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:50.883823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:50.881424    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"
Oct 29 23:01:52.252012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.251981    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"]
Oct 29 23:01:52.253074 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.253056    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1804496-00d1-4679-a069-89a1caec4e75" containerName="istio-proxy"
Oct 29 23:01:52.256219 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.256202    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.281619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.281588    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"]
Oct 29 23:01:52.318206 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.318172    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb3d37c-5657-4075-843c-be4d358e33f9-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.318395 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.318332    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg85s\" (UniqueName: \"kubernetes.io/projected/efb3d37c-5657-4075-843c-be4d358e33f9-kube-api-access-vg85s\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.318592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.318560    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.318669 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.318605    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb3d37c-5657-4075-843c-be4d358e33f9-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.318794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.318772    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.318923 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.318903    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb3d37c-5657-4075-843c-be4d358e33f9-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.318988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.318955    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.319050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.318998    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.319104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.319074    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.420496 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.420464    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.420655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.420552    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb3d37c-5657-4075-843c-be4d358e33f9-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.420655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.420581    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.420655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.420610    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.420811 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.420661    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.420811 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.420700    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb3d37c-5657-4075-843c-be4d358e33f9-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.421768 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.420845    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg85s\" (UniqueName: \"kubernetes.io/projected/efb3d37c-5657-4075-843c-be4d358e33f9-kube-api-access-vg85s\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.421768 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.420890    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.421768 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.421044    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.421768 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.421062    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.421768 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.421077    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb3d37c-5657-4075-843c-be4d358e33f9-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.421768 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.421085    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.421768 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.421315    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.421768 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.421735    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb3d37c-5657-4075-843c-be4d358e33f9-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.423626 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.422996    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.423626 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.423082    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb3d37c-5657-4075-843c-be4d358e33f9-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.442886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.442859    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb3d37c-5657-4075-843c-be4d358e33f9-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.443008 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.442982    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg85s\" (UniqueName: \"kubernetes.io/projected/efb3d37c-5657-4075-843c-be4d358e33f9-kube-api-access-vg85s\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.573564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.573478    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:52.912976 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.912947    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"]
Oct 29 23:01:52.914420 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:01:52.914393    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb3d37c_5657_4075_843c_be4d358e33f9.slice/crio-9444f26e6784f35bc438fd3cfbd69e9e60532671a63197c0c151e70b23537b73 WatchSource:0}: Error finding container 9444f26e6784f35bc438fd3cfbd69e9e60532671a63197c0c151e70b23537b73: Status 404 returned error can't find the container with id 9444f26e6784f35bc438fd3cfbd69e9e60532671a63197c0c151e70b23537b73
Oct 29 23:01:52.917767 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.917731    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:01:52.917915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.917819    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:01:52.917915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:52.917854    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:01:53.001850 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:53.001825    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:53.930952 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:53.930911    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t" event={"ID":"efb3d37c-5657-4075-843c-be4d358e33f9","Type":"ContainerStarted","Data":"3fee0c0055d2602dce74c528b08d3a6f61ee4283e70d57417fde574852eaab8b"}
Oct 29 23:01:53.931504 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:53.930972    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t" event={"ID":"efb3d37c-5657-4075-843c-be4d358e33f9","Type":"ContainerStarted","Data":"9444f26e6784f35bc438fd3cfbd69e9e60532671a63197c0c151e70b23537b73"}
Oct 29 23:01:53.999401 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:53.999342    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t" podStartSLOduration=1.999326585 podStartE2EDuration="1.999326585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:53.984903528 +0000 UTC m=+1117.327852276" watchObservedRunningTime="2026-10-29 23:01:53.999326585 +0000 UTC m=+1117.342275329"
Oct 29 23:01:54.577826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:54.577680    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:54.581824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:54.579524    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 29 23:01:54.581824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:54.579597    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t" podUID="efb3d37c-5657-4075-843c-be4d358e33f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 29 23:01:55.575822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:55.575153    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 29 23:01:55.575822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:55.575214    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t" podUID="efb3d37c-5657-4075-843c-be4d358e33f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 29 23:01:55.735168 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:55.733249    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:56.580239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:56.580198    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 29 23:01:56.580829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:56.580773    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t" podUID="efb3d37c-5657-4075-843c-be4d358e33f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 29 23:01:57.584363 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:57.584327    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 29 23:01:57.585968 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:57.584904    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t" podUID="efb3d37c-5657-4075-843c-be4d358e33f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 29 23:01:58.015760 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:58.015717    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:58.175291 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:58.175251    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:58.318905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:58.318763    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:58.584009 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:58.583914    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 29 23:01:58.584009 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:58.583979    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t" podUID="efb3d37c-5657-4075-843c-be4d358e33f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 29 23:01:59.592479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:59.592444    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:59.592969 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:59.592935    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:01:59.595323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:01:59.595292    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:02:00.567424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:00.567390    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:01.587536 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:01.587499    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:02.183764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:02.183730    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:02.653704 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:02.653660    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:02.832044 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:02.831078    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:03.185771 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:03.185725    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:05.088678 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:05.088644    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:06.890065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:06.890025    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:07.015081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.015042    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:07.144744 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.144620    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:07.407472 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.407368    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"]
Oct 29 23:02:07.414300 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.414265    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.462960 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.454825    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"]
Oct 29 23:02:07.515909 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.515877    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.516108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.515934    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.516108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.516056    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.516108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.516089    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.516306 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.516109    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snsm7\" (UniqueName: \"kubernetes.io/projected/a8f4481b-afbf-41b9-b739-df14b6a3e77f-kube-api-access-snsm7\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.516306 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.516196    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.516306 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.516226    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.516306 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.516295    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.517711 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.516379    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.618058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.618021    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.618240 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.618097    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.618240 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.618170    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.618240 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.618230    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.618927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.618261    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snsm7\" (UniqueName: \"kubernetes.io/projected/a8f4481b-afbf-41b9-b739-df14b6a3e77f-kube-api-access-snsm7\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.618927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.618455    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.618927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.618570    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-credential-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.618927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.618653    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.618927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.618729    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-data\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.618927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.618864    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-workload-certs\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.618927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.618871    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.620887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.618999    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.620887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.619294    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-workload-socket\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.620887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.619619    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istiod-ca-cert\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.628907 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.628884    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-envoy\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.629212 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.629189    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-podinfo\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.642915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.642888    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-token\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.643170 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.643147    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snsm7\" (UniqueName: \"kubernetes.io/projected/a8f4481b-afbf-41b9-b739-df14b6a3e77f-kube-api-access-snsm7\") pod \"e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") " pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:07.758381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:07.758298    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:08.114918 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:08.114884    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"]
Oct 29 23:02:08.121973 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:02:08.121912    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f4481b_afbf_41b9_b739_df14b6a3e77f.slice/crio-ea0d438abb530f6ed1d42f8ea6db5d149c7897c8daf3bb83d7d33d9d71cea23f WatchSource:0}: Error finding container ea0d438abb530f6ed1d42f8ea6db5d149c7897c8daf3bb83d7d33d9d71cea23f: Status 404 returned error can't find the container with id ea0d438abb530f6ed1d42f8ea6db5d149c7897c8daf3bb83d7d33d9d71cea23f
Oct 29 23:02:08.126592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:08.126545    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:02:08.127342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:08.126649    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:02:08.127342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:08.126698    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:02:08.588606 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:08.588478    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:08.714976 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:08.714930    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:09.209991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:09.209266    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq" event={"ID":"a8f4481b-afbf-41b9-b739-df14b6a3e77f","Type":"ContainerStarted","Data":"25bc92ffd492b08ef1b794e443d4c8214242944a2692f6a63fe05a7d5791565e"}
Oct 29 23:02:09.210419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:09.210027    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq" event={"ID":"a8f4481b-afbf-41b9-b739-df14b6a3e77f","Type":"ContainerStarted","Data":"ea0d438abb530f6ed1d42f8ea6db5d149c7897c8daf3bb83d7d33d9d71cea23f"}
Oct 29 23:02:09.283079 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:09.283024    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq" podStartSLOduration=2.283008967 podStartE2EDuration="2.283008967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:09.266429436 +0000 UTC m=+1132.609378180" watchObservedRunningTime="2026-10-29 23:02:09.283008967 +0000 UTC m=+1132.625957711"
Oct 29 23:02:09.762825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:09.761567    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:09.762825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:09.762327    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 29 23:02:09.762825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:09.762441    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq" podUID="a8f4481b-afbf-41b9-b739-df14b6a3e77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 29 23:02:10.072535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:10.072437    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:10.765558 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:10.762190    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 29 23:02:10.765558 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:10.762265    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq" podUID="a8f4481b-afbf-41b9-b739-df14b6a3e77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 29 23:02:11.285825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:11.284703    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:11.418828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:11.418038    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:11.774053 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:11.767039    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 29 23:02:11.774053 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:11.767106    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq" podUID="a8f4481b-afbf-41b9-b739-df14b6a3e77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 29 23:02:12.585472 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:12.585435    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:12.783854 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:12.782612    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 29 23:02:12.783854 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:12.782674    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq" podUID="a8f4481b-afbf-41b9-b739-df14b6a3e77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 29 23:02:13.317980 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:13.317945    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:13.772834 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:13.758847    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 29 23:02:13.772834 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:13.758907    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq" podUID="a8f4481b-afbf-41b9-b739-df14b6a3e77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 29 23:02:14.385108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:14.385068    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:14.787262 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:14.786236    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 29 23:02:14.787262 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:14.786306    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq" podUID="a8f4481b-afbf-41b9-b739-df14b6a3e77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 29 23:02:15.764267 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:15.764238    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:15.764701 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:15.764503    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:15.765081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:15.765060    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:02:16.121144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:16.121113    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:17.420793 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:02:17.420745    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:02:17.887628 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:17.887526    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:18.006582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:18.006544    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:18.127566 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:18.127527    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:19.668770 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:19.668724    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:21.300008 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:21.299329    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"]
Oct 29 23:02:21.308042 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:21.304846    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j" podUID="0a1a7c9a-d2ea-492a-9324-3596700b59d2" containerName="maas-api" containerID="cri-o://f6126a72de5e71e6b6db3c1a5dc6b11224cadf8f4dd70b0eb4f5d325b70a4225" gracePeriod=30
Oct 29 23:02:21.381171 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:21.361438    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:22.095698 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.089849    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"
Oct 29 23:02:22.161951 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.156408    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1a7c9a-d2ea-492a-9324-3596700b59d2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a1a7c9a-d2ea-492a-9324-3596700b59d2-openshift-service-ca\") pod \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\" (UID: \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\") "
Oct 29 23:02:22.161951 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.156547    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1a7c9a-d2ea-492a-9324-3596700b59d2-kube-api-access-2cdxl\" (UniqueName: \"kubernetes.io/projected/0a1a7c9a-d2ea-492a-9324-3596700b59d2-kube-api-access-2cdxl\") pod \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\" (UID: \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\") "
Oct 29 23:02:22.161951 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.156618    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a1a7c9a-d2ea-492a-9324-3596700b59d2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a1a7c9a-d2ea-492a-9324-3596700b59d2-maas-api-tls\") pod \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\" (UID: \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\") "
Oct 29 23:02:22.161951 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.156845    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a1a7c9a-d2ea-492a-9324-3596700b59d2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a1a7c9a-d2ea-492a-9324-3596700b59d2-metrics-certs\") pod \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\" (UID: \"0a1a7c9a-d2ea-492a-9324-3596700b59d2\") "
Oct 29 23:02:22.179833 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.172190    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1a7c9a-d2ea-492a-9324-3596700b59d2-openshift-service-ca" pod "0a1a7c9a-d2ea-492a-9324-3596700b59d2" (UID: "0a1a7c9a-d2ea-492a-9324-3596700b59d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:22.254155 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.250136    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:22.270598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.270533    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a1a7c9a-d2ea-492a-9324-3596700b59d2-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:22.331132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.331082    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a1a7c9a-d2ea-492a-9324-3596700b59d2-metrics-certs" pod "0a1a7c9a-d2ea-492a-9324-3596700b59d2" (UID: "0a1a7c9a-d2ea-492a-9324-3596700b59d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:22.336853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.336753    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1a7c9a-d2ea-492a-9324-3596700b59d2-kube-api-access-2cdxl" pod "0a1a7c9a-d2ea-492a-9324-3596700b59d2" (UID: "0a1a7c9a-d2ea-492a-9324-3596700b59d2"). InnerVolumeSpecName "kube-api-access-2cdxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:22.342828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.340017    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a1a7c9a-d2ea-492a-9324-3596700b59d2-maas-api-tls" pod "0a1a7c9a-d2ea-492a-9324-3596700b59d2" (UID: "0a1a7c9a-d2ea-492a-9324-3596700b59d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:22.397562 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.377082    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cdxl\" (UniqueName: \"kubernetes.io/projected/0a1a7c9a-d2ea-492a-9324-3596700b59d2-kube-api-access-2cdxl\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:22.397562 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.377117    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a1a7c9a-d2ea-492a-9324-3596700b59d2-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:22.397562 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.377132    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a1a7c9a-d2ea-492a-9324-3596700b59d2-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:22.470790 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.470700    5428 generic.go:356] "Generic (PLEG): container finished" podID="0a1a7c9a-d2ea-492a-9324-3596700b59d2" containerID="f6126a72de5e71e6b6db3c1a5dc6b11224cadf8f4dd70b0eb4f5d325b70a4225" exitCode=0
Oct 29 23:02:22.471064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.470738    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"
Oct 29 23:02:22.471306 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.470755    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j" event={"ID":"0a1a7c9a-d2ea-492a-9324-3596700b59d2","Type":"ContainerDied","Data":"f6126a72de5e71e6b6db3c1a5dc6b11224cadf8f4dd70b0eb4f5d325b70a4225"}
Oct 29 23:02:22.471449 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.471433    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j" event={"ID":"0a1a7c9a-d2ea-492a-9324-3596700b59d2","Type":"ContainerDied","Data":"725369ea985b2061252833465e9f53384bff388cc9c6dd0d9a5f479692040888"}
Oct 29 23:02:22.471561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.471548    5428 scope.go:122] "RemoveContainer" containerID="f6126a72de5e71e6b6db3c1a5dc6b11224cadf8f4dd70b0eb4f5d325b70a4225"
Oct 29 23:02:22.491029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.490999    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:22.531956 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.531923    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"]
Oct 29 23:02:22.536401 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.536377    5428 scope.go:122] "RemoveContainer" containerID="f6126a72de5e71e6b6db3c1a5dc6b11224cadf8f4dd70b0eb4f5d325b70a4225"
Oct 29 23:02:22.537360 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:02:22.537332    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6126a72de5e71e6b6db3c1a5dc6b11224cadf8f4dd70b0eb4f5d325b70a4225\": container with ID starting with f6126a72de5e71e6b6db3c1a5dc6b11224cadf8f4dd70b0eb4f5d325b70a4225 not found: ID does not exist" containerID="f6126a72de5e71e6b6db3c1a5dc6b11224cadf8f4dd70b0eb4f5d325b70a4225"
Oct 29 23:02:22.537500 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.537366    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6126a72de5e71e6b6db3c1a5dc6b11224cadf8f4dd70b0eb4f5d325b70a4225"} err="failed to get container status \"f6126a72de5e71e6b6db3c1a5dc6b11224cadf8f4dd70b0eb4f5d325b70a4225\": rpc error: code = NotFound desc = could not find container \"f6126a72de5e71e6b6db3c1a5dc6b11224cadf8f4dd70b0eb4f5d325b70a4225\": container with ID starting with f6126a72de5e71e6b6db3c1a5dc6b11224cadf8f4dd70b0eb4f5d325b70a4225 not found: ID does not exist"
Oct 29 23:02:22.538652 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.538628    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-479c2a-58fc846b79-dlw2j"]
Oct 29 23:02:22.649909 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.649872    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:22.812072 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:22.811983    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a1a7c9a-d2ea-492a-9324-3596700b59d2" path="/var/lib/kubelet/pods/0a1a7c9a-d2ea-492a-9324-3596700b59d2/volumes"
Oct 29 23:02:23.076844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:23.076740    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:25.392301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:25.392263    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:25.981957 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:25.981926    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"]
Oct 29 23:02:25.983289 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:25.983269    5428 cpu_manager.go:414] "containerMap: removing container" podUID="0a1a7c9a-d2ea-492a-9324-3596700b59d2" containerName="maas-api"
Oct 29 23:02:25.983366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:25.983292    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1a7c9a-d2ea-492a-9324-3596700b59d2" containerName="maas-api"
Oct 29 23:02:25.983478 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:25.983467    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1a7c9a-d2ea-492a-9324-3596700b59d2" containerName="maas-api"
Oct 29 23:02:25.987223 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:25.987201    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:02:26.002940 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.002919    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-44a4e4-serving-cert\""
Oct 29 23:02:26.022298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.022268    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"]
Oct 29 23:02:26.100742 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.100702    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7242d814-03a8-4079-9202-ede3445634b3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt\" (UID: \"7242d814-03a8-4079-9202-ede3445634b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:02:26.100910 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.100879    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwj87\" (UniqueName: \"kubernetes.io/projected/7242d814-03a8-4079-9202-ede3445634b3-kube-api-access-dwj87\") pod \"maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt\" (UID: \"7242d814-03a8-4079-9202-ede3445634b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:02:26.101114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.101090    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7242d814-03a8-4079-9202-ede3445634b3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt\" (UID: \"7242d814-03a8-4079-9202-ede3445634b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:02:26.101198 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.101181    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7242d814-03a8-4079-9202-ede3445634b3-metrics-certs\") pod \"maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt\" (UID: \"7242d814-03a8-4079-9202-ede3445634b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:02:26.134705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.134662    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:26.203233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.203206    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwj87\" (UniqueName: \"kubernetes.io/projected/7242d814-03a8-4079-9202-ede3445634b3-kube-api-access-dwj87\") pod \"maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt\" (UID: \"7242d814-03a8-4079-9202-ede3445634b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:02:26.203965 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.203341    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7242d814-03a8-4079-9202-ede3445634b3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt\" (UID: \"7242d814-03a8-4079-9202-ede3445634b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:02:26.203965 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.203382    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7242d814-03a8-4079-9202-ede3445634b3-metrics-certs\") pod \"maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt\" (UID: \"7242d814-03a8-4079-9202-ede3445634b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:02:26.203965 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.203519    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7242d814-03a8-4079-9202-ede3445634b3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt\" (UID: \"7242d814-03a8-4079-9202-ede3445634b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:02:26.203965 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:02:26.203672    5428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-serving-cert: secret "maas-api-e2e-ipp-a-44a4e4-serving-cert" not found
Oct 29 23:02:26.203965 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:02:26.203776    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7242d814-03a8-4079-9202-ede3445634b3-maas-api-tls podName:7242d814-03a8-4079-9202-ede3445634b3 nodeName:}" failed. No retries permitted until 2026-10-29 23:02:26.703727628 +0000 UTC m=+1150.046676368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7242d814-03a8-4079-9202-ede3445634b3-maas-api-tls") pod "maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt" (UID: "7242d814-03a8-4079-9202-ede3445634b3") : secret "maas-api-e2e-ipp-a-44a4e4-serving-cert" not found
Oct 29 23:02:26.205155 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.204118    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7242d814-03a8-4079-9202-ede3445634b3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt\" (UID: \"7242d814-03a8-4079-9202-ede3445634b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:02:26.207147 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.207129    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7242d814-03a8-4079-9202-ede3445634b3-metrics-certs\") pod \"maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt\" (UID: \"7242d814-03a8-4079-9202-ede3445634b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:02:26.228754 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.228728    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwj87\" (UniqueName: \"kubernetes.io/projected/7242d814-03a8-4079-9202-ede3445634b3-kube-api-access-dwj87\") pod \"maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt\" (UID: \"7242d814-03a8-4079-9202-ede3445634b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:02:26.272879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.269839    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:26.405213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.405179    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:26.711992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.711955    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7242d814-03a8-4079-9202-ede3445634b3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt\" (UID: \"7242d814-03a8-4079-9202-ede3445634b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:02:26.715197 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.715169    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7242d814-03a8-4079-9202-ede3445634b3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt\" (UID: \"7242d814-03a8-4079-9202-ede3445634b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:02:26.902427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:26.902396    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:02:27.272738 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:27.272710    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"]
Oct 29 23:02:27.275070 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:02:27.275043    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7242d814_03a8_4079_9202_ede3445634b3.slice/crio-121e56c803dc8ab8464e2d3b3723f6facc6c7b8a79df86ecaeaaf0b3eaf2f4de WatchSource:0}: Error finding container 121e56c803dc8ab8464e2d3b3723f6facc6c7b8a79df86ecaeaaf0b3eaf2f4de: Status 404 returned error can't find the container with id 121e56c803dc8ab8464e2d3b3723f6facc6c7b8a79df86ecaeaaf0b3eaf2f4de
Oct 29 23:02:27.532317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:27.532235    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt" event={"ID":"7242d814-03a8-4079-9202-ede3445634b3","Type":"ContainerStarted","Data":"121e56c803dc8ab8464e2d3b3723f6facc6c7b8a79df86ecaeaaf0b3eaf2f4de"}
Oct 29 23:02:28.542013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:28.541976    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt" event={"ID":"7242d814-03a8-4079-9202-ede3445634b3","Type":"ContainerStarted","Data":"eb4ffe848a4dba35e05085e838ef329d7e3286f5a631593b2915c787f59613f5"}
Oct 29 23:02:28.542443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:28.542041    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:02:28.586144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:28.586080    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt" podStartSLOduration=3.210972099 podStartE2EDuration="3.58606667s" totalImagesPullingTime="375.094571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:28.584174953 +0000 UTC m=+1151.927123699" watchObservedRunningTime="2026-10-29 23:02:28.58606667 +0000 UTC m=+1151.929015451"
Oct 29 23:02:29.280906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:29.280867    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:29.407774 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:29.407735    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:29.524849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:29.524779    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:29.849127 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:29.849085    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"]
Oct 29 23:02:29.849697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:29.849354    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9" podUID="d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" containerName="istio-proxy" containerID="cri-o://9de25fb15a301b9dd0b813455139e1c5037e612081d5f9798343ec99110148eb" gracePeriod=30
Oct 29 23:02:29.891927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:29.891827    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:31.484949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:31.484918    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"]
Oct 29 23:02:31.488268 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:31.488237    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf" podUID="71d90c5f-4afa-43f1-8574-4e271260f1c3" containerName="maas-api" containerID="cri-o://6ccf3271cf2b0bef3e9f15a51ce3863aaccc93dc778d32d76d578bfad5ab63b1" gracePeriod=30
Oct 29 23:02:31.573499 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:31.573465    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:31.841191 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:31.841137    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"
Oct 29 23:02:31.944732 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:31.944690    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d90c5f-4afa-43f1-8574-4e271260f1c3-kube-api-access-l7t62\" (UniqueName: \"kubernetes.io/projected/71d90c5f-4afa-43f1-8574-4e271260f1c3-kube-api-access-l7t62\") pod \"71d90c5f-4afa-43f1-8574-4e271260f1c3\" (UID: \"71d90c5f-4afa-43f1-8574-4e271260f1c3\") "
Oct 29 23:02:31.945915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:31.944922    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71d90c5f-4afa-43f1-8574-4e271260f1c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d90c5f-4afa-43f1-8574-4e271260f1c3-metrics-certs\") pod \"71d90c5f-4afa-43f1-8574-4e271260f1c3\" (UID: \"71d90c5f-4afa-43f1-8574-4e271260f1c3\") "
Oct 29 23:02:31.945915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:31.945124    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d90c5f-4afa-43f1-8574-4e271260f1c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71d90c5f-4afa-43f1-8574-4e271260f1c3-openshift-service-ca\") pod \"71d90c5f-4afa-43f1-8574-4e271260f1c3\" (UID: \"71d90c5f-4afa-43f1-8574-4e271260f1c3\") "
Oct 29 23:02:31.945915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:31.945202    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71d90c5f-4afa-43f1-8574-4e271260f1c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71d90c5f-4afa-43f1-8574-4e271260f1c3-maas-api-tls\") pod \"71d90c5f-4afa-43f1-8574-4e271260f1c3\" (UID: \"71d90c5f-4afa-43f1-8574-4e271260f1c3\") "
Oct 29 23:02:31.947424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:31.947350    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d90c5f-4afa-43f1-8574-4e271260f1c3-openshift-service-ca" pod "71d90c5f-4afa-43f1-8574-4e271260f1c3" (UID: "71d90c5f-4afa-43f1-8574-4e271260f1c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:31.950535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:31.950427    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d90c5f-4afa-43f1-8574-4e271260f1c3-kube-api-access-l7t62" pod "71d90c5f-4afa-43f1-8574-4e271260f1c3" (UID: "71d90c5f-4afa-43f1-8574-4e271260f1c3"). InnerVolumeSpecName "kube-api-access-l7t62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:31.950883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:31.950854    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d90c5f-4afa-43f1-8574-4e271260f1c3-maas-api-tls" pod "71d90c5f-4afa-43f1-8574-4e271260f1c3" (UID: "71d90c5f-4afa-43f1-8574-4e271260f1c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:31.951913 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:31.951889    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d90c5f-4afa-43f1-8574-4e271260f1c3-metrics-certs" pod "71d90c5f-4afa-43f1-8574-4e271260f1c3" (UID: "71d90c5f-4afa-43f1-8574-4e271260f1c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:32.049359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:32.049265    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71d90c5f-4afa-43f1-8574-4e271260f1c3-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:32.049359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:32.049307    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71d90c5f-4afa-43f1-8574-4e271260f1c3-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:32.049359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:32.049324    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71d90c5f-4afa-43f1-8574-4e271260f1c3-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:32.049359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:32.049337    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7t62\" (UniqueName: \"kubernetes.io/projected/71d90c5f-4afa-43f1-8574-4e271260f1c3-kube-api-access-l7t62\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:32.584878 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:32.584790    5428 generic.go:356] "Generic (PLEG): container finished" podID="71d90c5f-4afa-43f1-8574-4e271260f1c3" containerID="6ccf3271cf2b0bef3e9f15a51ce3863aaccc93dc778d32d76d578bfad5ab63b1" exitCode=0
Oct 29 23:02:32.584878 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:32.584853    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf" event={"ID":"71d90c5f-4afa-43f1-8574-4e271260f1c3","Type":"ContainerDied","Data":"6ccf3271cf2b0bef3e9f15a51ce3863aaccc93dc778d32d76d578bfad5ab63b1"}
Oct 29 23:02:32.584878 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:32.584877    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"
Oct 29 23:02:32.585594 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:32.585016    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf" event={"ID":"71d90c5f-4afa-43f1-8574-4e271260f1c3","Type":"ContainerDied","Data":"f3164685e6664fbdcb4594477235c4ea354cf4b1fed19bc3ae276459eeb95667"}
Oct 29 23:02:32.585594 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:32.585061    5428 scope.go:122] "RemoveContainer" containerID="6ccf3271cf2b0bef3e9f15a51ce3863aaccc93dc778d32d76d578bfad5ab63b1"
Oct 29 23:02:32.603234 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:32.603210    5428 scope.go:122] "RemoveContainer" containerID="6ccf3271cf2b0bef3e9f15a51ce3863aaccc93dc778d32d76d578bfad5ab63b1"
Oct 29 23:02:32.603668 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:02:32.603633    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ccf3271cf2b0bef3e9f15a51ce3863aaccc93dc778d32d76d578bfad5ab63b1\": container with ID starting with 6ccf3271cf2b0bef3e9f15a51ce3863aaccc93dc778d32d76d578bfad5ab63b1 not found: ID does not exist" containerID="6ccf3271cf2b0bef3e9f15a51ce3863aaccc93dc778d32d76d578bfad5ab63b1"
Oct 29 23:02:32.603788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:32.603683    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ccf3271cf2b0bef3e9f15a51ce3863aaccc93dc778d32d76d578bfad5ab63b1"} err="failed to get container status \"6ccf3271cf2b0bef3e9f15a51ce3863aaccc93dc778d32d76d578bfad5ab63b1\": rpc error: code = NotFound desc = could not find container \"6ccf3271cf2b0bef3e9f15a51ce3863aaccc93dc778d32d76d578bfad5ab63b1\": container with ID starting with 6ccf3271cf2b0bef3e9f15a51ce3863aaccc93dc778d32d76d578bfad5ab63b1 not found: ID does not exist"
Oct 29 23:02:32.644391 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:32.644361    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"]
Oct 29 23:02:32.649039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:32.649017    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79e6c631-6d9c4cf546-h7vgf"]
Oct 29 23:02:32.779894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:32.779853    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d90c5f-4afa-43f1-8574-4e271260f1c3" path="/var/lib/kubelet/pods/71d90c5f-4afa-43f1-8574-4e271260f1c3/volumes"
Oct 29 23:02:33.459625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:33.459370    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:33.584449 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:33.584409    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:33.723679 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:33.723589    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:34.155063 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:34.155031    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:34.557737 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:34.557663    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:02:35.167708 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.167682    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 23:02:35.322689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.322657    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-credential-socket\") pod \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") "
Oct 29 23:02:35.322902 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.322750    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-podinfo\") pod \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") "
Oct 29 23:02:35.322902 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.322845    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-envoy\") pod \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") "
Oct 29 23:02:35.322902 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.322884    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-workload-socket\") pod \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") "
Oct 29 23:02:35.323087 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.322905    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-data\") pod \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") "
Oct 29 23:02:35.323087 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.322936    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-credential-socket" pod "d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" (UID: "d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:35.323087 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.323060    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-kube-api-access-25gql\" (UniqueName: \"kubernetes.io/projected/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-kube-api-access-25gql\") pod \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") "
Oct 29 23:02:35.323919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.323082    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-token\" (UniqueName: \"kubernetes.io/projected/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-token\") pod \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") "
Oct 29 23:02:35.323919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.323142    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-data" pod "d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" (UID: "d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:35.323919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.323315    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istiod-ca-cert\") pod \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") "
Oct 29 23:02:35.323919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.323368    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-workload-certs\") pod \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\" (UID: \"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763\") "
Oct 29 23:02:35.323919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.323588    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-workload-socket" pod "d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" (UID: "d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:35.323919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.323704    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-workload-certs" pod "d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" (UID: "d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:35.324899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.324297    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.324899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.324316    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.324899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.324325    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.324899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.324334    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.324899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.324466    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istiod-ca-cert" pod "d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" (UID: "d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:35.325893 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.325871    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-envoy" pod "d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" (UID: "d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:35.325967 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.325898    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-token" pod "d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" (UID: "d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:35.326357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.326333    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-podinfo" pod "d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" (UID: "d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:35.326357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.326346    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-kube-api-access-25gql" pod "d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" (UID: "d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763"). InnerVolumeSpecName "kube-api-access-25gql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:35.426004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.425930    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.426004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.425957    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.426004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.425966    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.426004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.425977    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25gql\" (UniqueName: \"kubernetes.io/projected/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-kube-api-access-25gql\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.426004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.425986    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:35.615991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.615956    5428 generic.go:356] "Generic (PLEG): container finished" podID="d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" containerID="9de25fb15a301b9dd0b813455139e1c5037e612081d5f9798343ec99110148eb" exitCode=0
Oct 29 23:02:35.616169 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.616005    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"
Oct 29 23:02:35.616169 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.616037    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9" event={"ID":"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763","Type":"ContainerDied","Data":"9de25fb15a301b9dd0b813455139e1c5037e612081d5f9798343ec99110148eb"}
Oct 29 23:02:35.616169 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.616089    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9" event={"ID":"d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763","Type":"ContainerDied","Data":"1058ce3cda78c3714930ee5801f935e0d2af6dd5e3e5f5c6c6c0e2bd182b1236"}
Oct 29 23:02:35.616169 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.616111    5428 scope.go:122] "RemoveContainer" containerID="9de25fb15a301b9dd0b813455139e1c5037e612081d5f9798343ec99110148eb"
Oct 29 23:02:35.661880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.661856    5428 scope.go:122] "RemoveContainer" containerID="27660b71d8a1de8e19b5f7b9b3431863b594c9aaa8e92e8fcf2aaef13c9bf66b"
Oct 29 23:02:35.676198 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.676134    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"]
Oct 29 23:02:35.676380 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.676348    5428 scope.go:122] "RemoveContainer" containerID="9de25fb15a301b9dd0b813455139e1c5037e612081d5f9798343ec99110148eb"
Oct 29 23:02:35.676718 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:02:35.676693    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9de25fb15a301b9dd0b813455139e1c5037e612081d5f9798343ec99110148eb\": container with ID starting with 9de25fb15a301b9dd0b813455139e1c5037e612081d5f9798343ec99110148eb not found: ID does not exist" containerID="9de25fb15a301b9dd0b813455139e1c5037e612081d5f9798343ec99110148eb"
Oct 29 23:02:35.676864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.676726    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de25fb15a301b9dd0b813455139e1c5037e612081d5f9798343ec99110148eb"} err="failed to get container status \"9de25fb15a301b9dd0b813455139e1c5037e612081d5f9798343ec99110148eb\": rpc error: code = NotFound desc = could not find container \"9de25fb15a301b9dd0b813455139e1c5037e612081d5f9798343ec99110148eb\": container with ID starting with 9de25fb15a301b9dd0b813455139e1c5037e612081d5f9798343ec99110148eb not found: ID does not exist"
Oct 29 23:02:35.676864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.676749    5428 scope.go:122] "RemoveContainer" containerID="27660b71d8a1de8e19b5f7b9b3431863b594c9aaa8e92e8fcf2aaef13c9bf66b"
Oct 29 23:02:35.677195 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:02:35.677160    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27660b71d8a1de8e19b5f7b9b3431863b594c9aaa8e92e8fcf2aaef13c9bf66b\": container with ID starting with 27660b71d8a1de8e19b5f7b9b3431863b594c9aaa8e92e8fcf2aaef13c9bf66b not found: ID does not exist" containerID="27660b71d8a1de8e19b5f7b9b3431863b594c9aaa8e92e8fcf2aaef13c9bf66b"
Oct 29 23:02:35.677356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.677304    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27660b71d8a1de8e19b5f7b9b3431863b594c9aaa8e92e8fcf2aaef13c9bf66b"} err="failed to get container status \"27660b71d8a1de8e19b5f7b9b3431863b594c9aaa8e92e8fcf2aaef13c9bf66b\": rpc error: code = NotFound desc = could not find container \"27660b71d8a1de8e19b5f7b9b3431863b594c9aaa8e92e8fcf2aaef13c9bf66b\": container with ID starting with 27660b71d8a1de8e19b5f7b9b3431863b594c9aaa8e92e8fcf2aaef13c9bf66b not found: ID does not exist"
Oct 29 23:02:35.692422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.692400    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-479c2a-openshift-default-7bfbf9cbf-5lfl9"]
Oct 29 23:02:35.705057 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.705028    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"]
Oct 29 23:02:35.706795 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.706770    5428 cpu_manager.go:414] "containerMap: removing container" podUID="71d90c5f-4afa-43f1-8574-4e271260f1c3" containerName="maas-api"
Oct 29 23:02:35.708166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.706812    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d90c5f-4afa-43f1-8574-4e271260f1c3" containerName="maas-api"
Oct 29 23:02:35.708166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.706890    5428 cpu_manager.go:414] "containerMap: removing container" podUID="d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" containerName="istio-proxy"
Oct 29 23:02:35.708166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.706898    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" containerName="istio-proxy"
Oct 29 23:02:35.708166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.707322    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" containerName="istio-proxy"
Oct 29 23:02:35.708166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.707350    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" containerName="istio-proxy"
Oct 29 23:02:35.708166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.707377    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d90c5f-4afa-43f1-8574-4e271260f1c3" containerName="maas-api"
Oct 29 23:02:35.713095 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.713067    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.732973 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.732939    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"]
Oct 29 23:02:35.800041 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.800007    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ec4f6d-openshift-default-dockercfg-rrkm9\""
Oct 29 23:02:35.834014 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.833972    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-workload-certs\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.834255 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.834086    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-workload-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.834255 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.834145    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-envoy\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.834255 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.834218    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-token\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.834393 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.834306    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istiod-ca-cert\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.834393 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.834355    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpqkl\" (UniqueName: \"kubernetes.io/projected/cdf8d152-fc6c-441a-89bf-0d02421e74c2-kube-api-access-mpqkl\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.834459 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.834404    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-data\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.834504 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.834450    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-credential-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.834655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.834642    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-podinfo\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.936372 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.936290    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-workload-certs\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.936372 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.936351    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-workload-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.936618 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.936382    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-envoy\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.936618 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.936465    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-token\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.936618 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.936529    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istiod-ca-cert\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.936618 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.936585    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqkl\" (UniqueName: \"kubernetes.io/projected/cdf8d152-fc6c-441a-89bf-0d02421e74c2-kube-api-access-mpqkl\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.936920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.936636    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-data\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.936920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.936671    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-credential-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.936920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.936838    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-workload-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.937092 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.937014    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-podinfo\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.937092 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.937041    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-data\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.937298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.937275    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-workload-certs\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.937412 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.937384    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-credential-socket\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.937486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.937439    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istiod-ca-cert\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.938775 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.938752    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-envoy\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.939588 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.939566    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-podinfo\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.959725 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.959700    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-token\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.959980 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.959956    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpqkl\" (UniqueName: \"kubernetes.io/projected/cdf8d152-fc6c-441a-89bf-0d02421e74c2-kube-api-access-mpqkl\") pod \"e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") " pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:35.971575 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:35.971546    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:36.054854 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:36.054823    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:36.408643 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:02:36.408615    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf8d152_fc6c_441a_89bf_0d02421e74c2.slice/crio-88bbcdf1822dded508210b6fe967ce920076764a2e729953e08fe57794b15bee WatchSource:0}: Error finding container 88bbcdf1822dded508210b6fe967ce920076764a2e729953e08fe57794b15bee: Status 404 returned error can't find the container with id 88bbcdf1822dded508210b6fe967ce920076764a2e729953e08fe57794b15bee
Oct 29 23:02:36.409013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:36.408672    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"]
Oct 29 23:02:36.412485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:36.412453    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:02:36.412562 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:36.412521    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:02:36.412562 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:36.412550    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:02:36.628482 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:36.628444    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt" event={"ID":"cdf8d152-fc6c-441a-89bf-0d02421e74c2","Type":"ContainerStarted","Data":"1aaa0eb1ed3bb0d26dc0e322a403cbe64e5279960b4cc50cff30dc1110eb18ca"}
Oct 29 23:02:36.628670 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:36.628510    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt" event={"ID":"cdf8d152-fc6c-441a-89bf-0d02421e74c2","Type":"ContainerStarted","Data":"88bbcdf1822dded508210b6fe967ce920076764a2e729953e08fe57794b15bee"}
Oct 29 23:02:36.679683 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:36.679570    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt" podStartSLOduration=1.679552185 podStartE2EDuration="1.679552185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:36.673736135 +0000 UTC m=+1160.016684879" watchObservedRunningTime="2026-10-29 23:02:36.679552185 +0000 UTC m=+1160.022500931"
Oct 29 23:02:36.782143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:36.782095    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" path="/var/lib/kubelet/pods/d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763/volumes"
Oct 29 23:02:37.055825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:37.055728    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:37.057107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:37.057080    5428 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 29 23:02:37.057482 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:37.057142    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt" podUID="cdf8d152-fc6c-441a-89bf-0d02421e74c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 29 23:02:38.056090 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:38.056056    5428 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 29 23:02:38.057739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:38.056103    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt" podUID="cdf8d152-fc6c-441a-89bf-0d02421e74c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 29 23:02:38.489762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:38.489723    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:39.056203 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:39.056166    5428 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 29 23:02:39.056606 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:39.056245    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt" podUID="cdf8d152-fc6c-441a-89bf-0d02421e74c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 29 23:02:39.109120 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:39.109087    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"]
Oct 29 23:02:39.236308 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:39.236275    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:40.055627 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:40.055586    5428 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 29 23:02:40.055841 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:40.055656    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt" podUID="cdf8d152-fc6c-441a-89bf-0d02421e74c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 29 23:02:40.265827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:40.264166    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:41.055919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:41.055878    5428 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 29 23:02:41.056115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:41.055953    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt" podUID="cdf8d152-fc6c-441a-89bf-0d02421e74c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 29 23:02:41.337547 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:41.337454    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:41.465847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:41.465788    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:42.055520 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:42.055477    5428 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 29 23:02:42.055730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:42.055535    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt" podUID="cdf8d152-fc6c-441a-89bf-0d02421e74c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 29 23:02:42.491040 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:42.490964    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:42.839495 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:42.839397    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v"]
Oct 29 23:02:42.952956 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:42.952917    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:43.068405 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:43.064355    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:43.068405 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:43.065167    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:43.068405 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:43.065888    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:43.068776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:43.068758    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:02:45.174718 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:45.174681    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:47.371115 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:02:47.371075    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:02:47.723695 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.723611    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"]
Oct 29 23:02:47.724572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.724552    5428 cpu_manager.go:414] "containerMap: removing container" podUID="d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" containerName="istio-proxy"
Oct 29 23:02:47.724650 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.724574    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0fa5ed2-1b9e-41e7-8d0d-e56a7c2f1763" containerName="istio-proxy"
Oct 29 23:02:47.729869 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.729851    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:02:47.770628 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.770576    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d629caba-fbff-4186-9a17-e2ad12edb908-openshift-service-ca\") pod \"maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:02:47.770628 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.770630    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-maas-api-tls\") pod \"maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:02:47.770906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.770845    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-metrics-certs\") pod \"maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:02:47.770976 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.770924    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7nj5\" (UniqueName: \"kubernetes.io/projected/d629caba-fbff-4186-9a17-e2ad12edb908-kube-api-access-l7nj5\") pod \"maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:02:47.795456 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.795428    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d430df-serving-cert\""
Oct 29 23:02:47.802930 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.802902    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"]
Oct 29 23:02:47.840030 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.839994    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v" podUID="a32d5c46-fd89-4d2e-a902-9c3c80abfcb7" containerName="payload-pre-processing" containerID="cri-o://844e1c998df1584e86fbf9725b96ed740da2b51d570e88c3108227c88fddd6d5" gracePeriod=25
Oct 29 23:02:47.872843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.872788    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7nj5\" (UniqueName: \"kubernetes.io/projected/d629caba-fbff-4186-9a17-e2ad12edb908-kube-api-access-l7nj5\") pod \"maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:02:47.873457 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.873430    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d629caba-fbff-4186-9a17-e2ad12edb908-openshift-service-ca\") pod \"maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:02:47.873611 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.873587    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-maas-api-tls\") pod \"maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:02:47.873860 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:02:47.873702    5428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-serving-cert: secret "maas-api-e2e-api-b-d430df-serving-cert" not found
Oct 29 23:02:47.873860 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:02:47.873779    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-maas-api-tls podName:d629caba-fbff-4186-9a17-e2ad12edb908 nodeName:}" failed. No retries permitted until 2026-10-29 23:02:48.373759644 +0000 UTC m=+1171.716708371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-maas-api-tls") pod "maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl" (UID: "d629caba-fbff-4186-9a17-e2ad12edb908") : secret "maas-api-e2e-api-b-d430df-serving-cert" not found
Oct 29 23:02:47.874064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.873930    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-metrics-certs\") pod \"maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:02:47.881402 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.874689    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d629caba-fbff-4186-9a17-e2ad12edb908-openshift-service-ca\") pod \"maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:02:47.881402 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.876909    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-metrics-certs\") pod \"maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:02:47.898237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.897884    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7nj5\" (UniqueName: \"kubernetes.io/projected/d629caba-fbff-4186-9a17-e2ad12edb908-kube-api-access-l7nj5\") pod \"maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:02:47.903140 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:47.903073    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:48.018751 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:48.018670    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:48.138640 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:48.138606    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:48.396684 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:48.396643    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-maas-api-tls\") pod \"maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:02:48.397217 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:02:48.397112    5428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-serving-cert: secret "maas-api-e2e-api-b-d430df-serving-cert" not found
Oct 29 23:02:48.397217 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:02:48.397181    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-maas-api-tls podName:d629caba-fbff-4186-9a17-e2ad12edb908 nodeName:}" failed. No retries permitted until 2026-10-29 23:02:49.397161997 +0000 UTC m=+1172.740110724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-maas-api-tls") pod "maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl" (UID: "d629caba-fbff-4186-9a17-e2ad12edb908") : secret "maas-api-e2e-api-b-d430df-serving-cert" not found
Oct 29 23:02:48.553514 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:48.553452    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v"
Oct 29 23:02:48.604309 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:48.604271    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a32d5c46-fd89-4d2e-a902-9c3c80abfcb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a32d5c46-fd89-4d2e-a902-9c3c80abfcb7-plugins-config-volume\") pod \"a32d5c46-fd89-4d2e-a902-9c3c80abfcb7\" (UID: \"a32d5c46-fd89-4d2e-a902-9c3c80abfcb7\") "
Oct 29 23:02:48.604485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:48.604422    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a32d5c46-fd89-4d2e-a902-9c3c80abfcb7-kube-api-access-9q6dd\" (UniqueName: \"kubernetes.io/projected/a32d5c46-fd89-4d2e-a902-9c3c80abfcb7-kube-api-access-9q6dd\") pod \"a32d5c46-fd89-4d2e-a902-9c3c80abfcb7\" (UID: \"a32d5c46-fd89-4d2e-a902-9c3c80abfcb7\") "
Oct 29 23:02:48.605032 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:48.604999    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a32d5c46-fd89-4d2e-a902-9c3c80abfcb7-plugins-config-volume" pod "a32d5c46-fd89-4d2e-a902-9c3c80abfcb7" (UID: "a32d5c46-fd89-4d2e-a902-9c3c80abfcb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:48.605410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:48.605389    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a32d5c46-fd89-4d2e-a902-9c3c80abfcb7-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.609786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:48.609757    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32d5c46-fd89-4d2e-a902-9c3c80abfcb7-kube-api-access-9q6dd" pod "a32d5c46-fd89-4d2e-a902-9c3c80abfcb7" (UID: "a32d5c46-fd89-4d2e-a902-9c3c80abfcb7"). InnerVolumeSpecName "kube-api-access-9q6dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:48.707101 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:48.707069    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q6dd\" (UniqueName: \"kubernetes.io/projected/a32d5c46-fd89-4d2e-a902-9c3c80abfcb7-kube-api-access-9q6dd\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:48.926199 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:48.926164    5428 generic.go:356] "Generic (PLEG): container finished" podID="a32d5c46-fd89-4d2e-a902-9c3c80abfcb7" containerID="844e1c998df1584e86fbf9725b96ed740da2b51d570e88c3108227c88fddd6d5" exitCode=0
Oct 29 23:02:48.926199 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:48.926207    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v" event={"ID":"a32d5c46-fd89-4d2e-a902-9c3c80abfcb7","Type":"ContainerDied","Data":"844e1c998df1584e86fbf9725b96ed740da2b51d570e88c3108227c88fddd6d5"}
Oct 29 23:02:48.926508 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:48.926228    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v"
Oct 29 23:02:48.926508 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:48.926247    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v" event={"ID":"a32d5c46-fd89-4d2e-a902-9c3c80abfcb7","Type":"ContainerDied","Data":"1cf4539ec8f438a4d8350ba55a0e50dcebd58ca072fa5a391d676ef6d8442b87"}
Oct 29 23:02:48.926508 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:48.926270    5428 scope.go:122] "RemoveContainer" containerID="844e1c998df1584e86fbf9725b96ed740da2b51d570e88c3108227c88fddd6d5"
Oct 29 23:02:48.939677 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:48.939655    5428 scope.go:122] "RemoveContainer" containerID="844e1c998df1584e86fbf9725b96ed740da2b51d570e88c3108227c88fddd6d5"
Oct 29 23:02:48.940063 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:02:48.940021    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"844e1c998df1584e86fbf9725b96ed740da2b51d570e88c3108227c88fddd6d5\": container with ID starting with 844e1c998df1584e86fbf9725b96ed740da2b51d570e88c3108227c88fddd6d5 not found: ID does not exist" containerID="844e1c998df1584e86fbf9725b96ed740da2b51d570e88c3108227c88fddd6d5"
Oct 29 23:02:48.940205 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:48.940096    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"844e1c998df1584e86fbf9725b96ed740da2b51d570e88c3108227c88fddd6d5"} err="failed to get container status \"844e1c998df1584e86fbf9725b96ed740da2b51d570e88c3108227c88fddd6d5\": rpc error: code = NotFound desc = could not find container \"844e1c998df1584e86fbf9725b96ed740da2b51d570e88c3108227c88fddd6d5\": container with ID starting with 844e1c998df1584e86fbf9725b96ed740da2b51d570e88c3108227c88fddd6d5 not found: ID does not exist"
Oct 29 23:02:48.994378 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:48.994344    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v"]
Oct 29 23:02:49.001522 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:49.001490    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-df82ee-9cb4f4847-pck5v"]
Oct 29 23:02:49.420783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:49.420746    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-maas-api-tls\") pod \"maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:02:49.421228 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:02:49.420928    5428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-serving-cert: secret "maas-api-e2e-api-b-d430df-serving-cert" not found
Oct 29 23:02:49.421228 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:02:49.421035    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-maas-api-tls podName:d629caba-fbff-4186-9a17-e2ad12edb908 nodeName:}" failed. No retries permitted until 2026-10-29 23:02:51.421019549 +0000 UTC m=+1174.763968272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-maas-api-tls") pod "maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl" (UID: "d629caba-fbff-4186-9a17-e2ad12edb908") : secret "maas-api-e2e-api-b-d430df-serving-cert" not found
Oct 29 23:02:49.800114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:49.800010    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:50.779658 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:50.779623    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a32d5c46-fd89-4d2e-a902-9c3c80abfcb7" path="/var/lib/kubelet/pods/a32d5c46-fd89-4d2e-a902-9c3c80abfcb7/volumes"
Oct 29 23:02:51.078756 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:51.078678    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:51.471212 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:51.471177    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-maas-api-tls\") pod \"maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:02:51.473862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:51.473833    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-maas-api-tls\") pod \"maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:02:51.647575 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:51.647544    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:02:51.977003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:51.976969    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"]
Oct 29 23:02:51.977563 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:02:51.977539    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd629caba_fbff_4186_9a17_e2ad12edb908.slice/crio-2170b87cd3551963e96870c9c7b5e4bb2b9996fa86abbd275d051ecffcb012cd WatchSource:0}: Error finding container 2170b87cd3551963e96870c9c7b5e4bb2b9996fa86abbd275d051ecffcb012cd: Status 404 returned error can't find the container with id 2170b87cd3551963e96870c9c7b5e4bb2b9996fa86abbd275d051ecffcb012cd
Oct 29 23:02:52.980001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:52.979964    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl" event={"ID":"d629caba-fbff-4186-9a17-e2ad12edb908","Type":"ContainerStarted","Data":"b0f8bb9334deb0078b7398e92abc67336ec4b50c98c43debd414fd65340e7ab8"}
Oct 29 23:02:52.980416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:52.980028    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl" event={"ID":"d629caba-fbff-4186-9a17-e2ad12edb908","Type":"ContainerStarted","Data":"2170b87cd3551963e96870c9c7b5e4bb2b9996fa86abbd275d051ecffcb012cd"}
Oct 29 23:02:52.980416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:52.980058    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:02:53.029214 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:53.029161    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl" podStartSLOduration=5.636251171 podStartE2EDuration="6.029146419s" totalImagesPullingTime="392.895248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:53.02763054 +0000 UTC m=+1176.370579286" watchObservedRunningTime="2026-10-29 23:02:53.029146419 +0000 UTC m=+1176.372095184"
Oct 29 23:02:53.809786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:53.809748    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:54.150083 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.150035    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj" podUID="63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" containerName="main" containerID="cri-o://0a02f567e0ab7225097b4e06602220c64b6b3d71019d650a5ad524421dc6caa8" gracePeriod=45
Oct 29 23:02:54.448108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.448079    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 23:02:54.540184 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.540152    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-home\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-home\") pod \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") "
Oct 29 23:02:54.540371 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.540257    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-tmp-dir\") pod \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") "
Oct 29 23:02:54.540371 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.540297    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-model-cache\") pod \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") "
Oct 29 23:02:54.540371 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.540335    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-kube-api-access-9b6xp\" (UniqueName: \"kubernetes.io/projected/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-kube-api-access-9b6xp\") pod \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") "
Oct 29 23:02:54.540533 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.540371    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-tls-certs\" (UniqueName: \"kubernetes.io/secret/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-tls-certs\") pod \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") "
Oct 29 23:02:54.540533 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.540505    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-dshm\") pod \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\" (UID: \"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db\") "
Oct 29 23:02:54.540533 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.540480    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-home" pod "63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" (UID: "63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:54.540718 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.540559    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-tmp-dir" pod "63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" (UID: "63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:54.540718 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.540575    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-model-cache" pod "63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" (UID: "63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:54.541453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.541422    5428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-home\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:54.541592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.541454    5428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-tmp-dir\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:54.541592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.541468    5428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-model-cache\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:54.543972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.543946    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-dshm" pod "63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" (UID: "63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:54.544444 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.544412    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-tls-certs" pod "63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" (UID: "63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:54.544565 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.544487    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-kube-api-access-9b6xp" pod "63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" (UID: "63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db"). InnerVolumeSpecName "kube-api-access-9b6xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:54.642942 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.642905    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b6xp\" (UniqueName: \"kubernetes.io/projected/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-kube-api-access-9b6xp\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:54.642942 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.642938    5428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-tls-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:54.642942 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.642947    5428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db-dshm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:02:54.999218 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.999178    5428 generic.go:356] "Generic (PLEG): container finished" podID="63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" containerID="0a02f567e0ab7225097b4e06602220c64b6b3d71019d650a5ad524421dc6caa8" exitCode=0
Oct 29 23:02:54.999394 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.999229    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"
Oct 29 23:02:54.999394 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.999263    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj" event={"ID":"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db","Type":"ContainerDied","Data":"0a02f567e0ab7225097b4e06602220c64b6b3d71019d650a5ad524421dc6caa8"}
Oct 29 23:02:54.999394 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.999312    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj" event={"ID":"63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db","Type":"ContainerDied","Data":"cefa52c7e4fe27efa247e02a478faed6e66be591c5fb93dfcc81180fd692865d"}
Oct 29 23:02:54.999394 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:54.999331    5428 scope.go:122] "RemoveContainer" containerID="0a02f567e0ab7225097b4e06602220c64b6b3d71019d650a5ad524421dc6caa8"
Oct 29 23:02:55.045276 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:55.045247    5428 scope.go:122] "RemoveContainer" containerID="0a02f567e0ab7225097b4e06602220c64b6b3d71019d650a5ad524421dc6caa8"
Oct 29 23:02:55.045630 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:02:55.045604    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a02f567e0ab7225097b4e06602220c64b6b3d71019d650a5ad524421dc6caa8\": container with ID starting with 0a02f567e0ab7225097b4e06602220c64b6b3d71019d650a5ad524421dc6caa8 not found: ID does not exist" containerID="0a02f567e0ab7225097b4e06602220c64b6b3d71019d650a5ad524421dc6caa8"
Oct 29 23:02:55.045685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:55.045640    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a02f567e0ab7225097b4e06602220c64b6b3d71019d650a5ad524421dc6caa8"} err="failed to get container status \"0a02f567e0ab7225097b4e06602220c64b6b3d71019d650a5ad524421dc6caa8\": rpc error: code = NotFound desc = could not find container \"0a02f567e0ab7225097b4e06602220c64b6b3d71019d650a5ad524421dc6caa8\": container with ID starting with 0a02f567e0ab7225097b4e06602220c64b6b3d71019d650a5ad524421dc6caa8 not found: ID does not exist"
Oct 29 23:02:55.052775 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:55.052749    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"]
Oct 29 23:02:55.065046 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:55.065024    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj"]
Oct 29 23:02:55.332233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:55.332144    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-79e6c631/e2e-lifecycle-model-79e6c631-kserve-659bd99ff7-l6mqj" podUID="63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:02:55.454722 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:55.454673    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:56.779652 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:56.779622    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" path="/var/lib/kubelet/pods/63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db/volumes"
Oct 29 23:02:57.210255 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:57.210185    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:58.884765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:58.884065    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:58.996747 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:02:58.996715    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:03:00.924098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:00.924058    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:02.813102 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:02.813072    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:03.174969 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:03.174933    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:03.300648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:03.300609    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:03.428518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:03.428413    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:05.246937 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:05.246897    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:07.440269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:07.440233    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:07.891312 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:07.891263    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"]
Oct 29 23:03:07.891512 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:07.891491    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm" podUID="b457bfe9-1262-4ba5-bd19-fbdfd0e14310" containerName="istio-proxy" containerID="cri-o://79aa1c26da7fd4420274106f637e173b8df7b354e5eb44e4389f34fe624abdb7" gracePeriod=30
Oct 29 23:03:07.904342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:07.904311    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"]
Oct 29 23:03:07.904583 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:07.904556    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk" podUID="16750225-7382-4170-8297-4def491a1ede" containerName="istio-proxy" containerID="cri-o://bbccb5b86a5de6419444b08228a2af060bcc498019b5069073ac8151670bcb34" gracePeriod=30
Oct 29 23:03:07.909844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:07.909795    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"]
Oct 29 23:03:07.910067 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:07.910020    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2" podUID="204e9634-b4e0-477d-8114-10fd7ecf0f72" containerName="istio-proxy" containerID="cri-o://5b1e1e76ebe8ca74d49290ba27328a8c300cfcadcde8e9612f550e05cfb9abc9" gracePeriod=30
Oct 29 23:03:09.398758 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:09.398714    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:10.914228 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:10.914190    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"]
Oct 29 23:03:10.914630 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:10.914385    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p" podUID="0855c417-df52-4f3d-8a49-bc3b1fb88a0f" containerName="istio-proxy" containerID="cri-o://dc0e9f6496bbd8ea4fd7921f803ad38eb4abd52b6bbab2c763eeaaeb55403585" gracePeriod=30
Oct 29 23:03:10.926689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:10.926652    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"]
Oct 29 23:03:10.926996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:10.926961    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f" podUID="5b094a72-da28-4b91-9dec-f6d9d55543b9" containerName="istio-proxy" containerID="cri-o://661e425d0576452c701691e0015301a911143ab9a231b2b454f1609588c7df87" gracePeriod=30
Oct 29 23:03:11.408571 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:11.408534    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:11.932625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:11.932589    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:12.066646 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:12.066615    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:13.369840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.369640    5428 generic.go:356] "Generic (PLEG): container finished" podID="16750225-7382-4170-8297-4def491a1ede" containerID="bbccb5b86a5de6419444b08228a2af060bcc498019b5069073ac8151670bcb34" exitCode=0
Oct 29 23:03:13.370347 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.369893    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk" event={"ID":"16750225-7382-4170-8297-4def491a1ede","Type":"ContainerDied","Data":"bbccb5b86a5de6419444b08228a2af060bcc498019b5069073ac8151670bcb34"}
Oct 29 23:03:13.374737 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.372491    5428 generic.go:356] "Generic (PLEG): container finished" podID="204e9634-b4e0-477d-8114-10fd7ecf0f72" containerID="5b1e1e76ebe8ca74d49290ba27328a8c300cfcadcde8e9612f550e05cfb9abc9" exitCode=0
Oct 29 23:03:13.374737 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.372638    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2" event={"ID":"204e9634-b4e0-477d-8114-10fd7ecf0f72","Type":"ContainerDied","Data":"5b1e1e76ebe8ca74d49290ba27328a8c300cfcadcde8e9612f550e05cfb9abc9"}
Oct 29 23:03:13.377764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.377734    5428 generic.go:356] "Generic (PLEG): container finished" podID="b457bfe9-1262-4ba5-bd19-fbdfd0e14310" containerID="79aa1c26da7fd4420274106f637e173b8df7b354e5eb44e4389f34fe624abdb7" exitCode=0
Oct 29 23:03:13.378178 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.378004    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm" event={"ID":"b457bfe9-1262-4ba5-bd19-fbdfd0e14310","Type":"ContainerDied","Data":"79aa1c26da7fd4420274106f637e173b8df7b354e5eb44e4389f34fe624abdb7"}
Oct 29 23:03:13.681189 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.681085    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 23:03:13.727634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.727548    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 23:03:13.789701 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.789660    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-istio-envoy\") pod \"16750225-7382-4170-8297-4def491a1ede\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") "
Oct 29 23:03:13.789911 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.789730    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-workload-certs\") pod \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") "
Oct 29 23:03:13.789911 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.789763    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16750225-7382-4170-8297-4def491a1ede-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16750225-7382-4170-8297-4def491a1ede-istio-podinfo\") pod \"16750225-7382-4170-8297-4def491a1ede\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") "
Oct 29 23:03:13.789911 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.789851    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-credential-socket\") pod \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") "
Oct 29 23:03:13.790087 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.789951    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-envoy\") pod \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") "
Oct 29 23:03:13.790087 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.789982    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-workload-socket\") pod \"16750225-7382-4170-8297-4def491a1ede\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") "
Oct 29 23:03:13.790087 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.790058    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istiod-ca-cert\") pod \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") "
Oct 29 23:03:13.790263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.790167    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-podinfo\") pod \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") "
Oct 29 23:03:13.790263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.790216    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16750225-7382-4170-8297-4def491a1ede-kube-api-access-t5rck\" (UniqueName: \"kubernetes.io/projected/16750225-7382-4170-8297-4def491a1ede-kube-api-access-t5rck\") pod \"16750225-7382-4170-8297-4def491a1ede\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") "
Oct 29 23:03:13.790263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.790254    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-credential-socket\") pod \"16750225-7382-4170-8297-4def491a1ede\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") "
Oct 29 23:03:13.790424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.790290    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-kube-api-access-bhjnw\" (UniqueName: \"kubernetes.io/projected/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-kube-api-access-bhjnw\") pod \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") "
Oct 29 23:03:13.790510 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.790485    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-istio-data\") pod \"16750225-7382-4170-8297-4def491a1ede\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") "
Oct 29 23:03:13.790583 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.790560    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-token\" (UniqueName: \"kubernetes.io/projected/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-token\") pod \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") "
Oct 29 23:03:13.790640 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.790631    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-workload-socket\") pod \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") "
Oct 29 23:03:13.790700 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.790664    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16750225-7382-4170-8297-4def491a1ede-istio-token\" (UniqueName: \"kubernetes.io/projected/16750225-7382-4170-8297-4def491a1ede-istio-token\") pod \"16750225-7382-4170-8297-4def491a1ede\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") "
Oct 29 23:03:13.790761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.790701    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-workload-certs\") pod \"16750225-7382-4170-8297-4def491a1ede\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") "
Oct 29 23:03:13.790761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.790740    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16750225-7382-4170-8297-4def491a1ede-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16750225-7382-4170-8297-4def491a1ede-istiod-ca-cert\") pod \"16750225-7382-4170-8297-4def491a1ede\" (UID: \"16750225-7382-4170-8297-4def491a1ede\") "
Oct 29 23:03:13.790924 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.790818    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-data\") pod \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\" (UID: \"b457bfe9-1262-4ba5-bd19-fbdfd0e14310\") "
Oct 29 23:03:13.809093 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.808928    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-data" pod "b457bfe9-1262-4ba5-bd19-fbdfd0e14310" (UID: "b457bfe9-1262-4ba5-bd19-fbdfd0e14310"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:13.810636 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.810570    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-workload-certs" pod "b457bfe9-1262-4ba5-bd19-fbdfd0e14310" (UID: "b457bfe9-1262-4ba5-bd19-fbdfd0e14310"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:13.818875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.810790    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-workload-socket" pod "16750225-7382-4170-8297-4def491a1ede" (UID: "16750225-7382-4170-8297-4def491a1ede"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:13.818875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.811743    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-istio-data" pod "16750225-7382-4170-8297-4def491a1ede" (UID: "16750225-7382-4170-8297-4def491a1ede"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:13.818875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.812373    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-credential-socket" pod "16750225-7382-4170-8297-4def491a1ede" (UID: "16750225-7382-4170-8297-4def491a1ede"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:13.828654 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.827070    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istiod-ca-cert" pod "b457bfe9-1262-4ba5-bd19-fbdfd0e14310" (UID: "b457bfe9-1262-4ba5-bd19-fbdfd0e14310"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:13.828654 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.828447    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-workload-certs" pod "16750225-7382-4170-8297-4def491a1ede" (UID: "16750225-7382-4170-8297-4def491a1ede"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:13.831008 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.830880    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-credential-socket" pod "b457bfe9-1262-4ba5-bd19-fbdfd0e14310" (UID: "b457bfe9-1262-4ba5-bd19-fbdfd0e14310"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:13.832001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.831947    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-workload-socket" pod "b457bfe9-1262-4ba5-bd19-fbdfd0e14310" (UID: "b457bfe9-1262-4ba5-bd19-fbdfd0e14310"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:13.832232 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.832181    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16750225-7382-4170-8297-4def491a1ede-istiod-ca-cert" pod "16750225-7382-4170-8297-4def491a1ede" (UID: "16750225-7382-4170-8297-4def491a1ede"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:13.842872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.842262    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-kube-api-access-bhjnw" pod "b457bfe9-1262-4ba5-bd19-fbdfd0e14310" (UID: "b457bfe9-1262-4ba5-bd19-fbdfd0e14310"). InnerVolumeSpecName "kube-api-access-bhjnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:13.852975 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.852932    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16750225-7382-4170-8297-4def491a1ede-istio-podinfo" pod "16750225-7382-4170-8297-4def491a1ede" (UID: "16750225-7382-4170-8297-4def491a1ede"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:13.854608 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.854554    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-podinfo" pod "b457bfe9-1262-4ba5-bd19-fbdfd0e14310" (UID: "b457bfe9-1262-4ba5-bd19-fbdfd0e14310"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:13.855117 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.855072    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16750225-7382-4170-8297-4def491a1ede-istio-token" pod "16750225-7382-4170-8297-4def491a1ede" (UID: "16750225-7382-4170-8297-4def491a1ede"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:13.855577 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.855535    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16750225-7382-4170-8297-4def491a1ede-kube-api-access-t5rck" pod "16750225-7382-4170-8297-4def491a1ede" (UID: "16750225-7382-4170-8297-4def491a1ede"). InnerVolumeSpecName "kube-api-access-t5rck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:13.855819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.855764    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-envoy" pod "b457bfe9-1262-4ba5-bd19-fbdfd0e14310" (UID: "b457bfe9-1262-4ba5-bd19-fbdfd0e14310"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:13.856038 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.855998    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-istio-envoy" pod "16750225-7382-4170-8297-4def491a1ede" (UID: "16750225-7382-4170-8297-4def491a1ede"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:13.858333 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.858298    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-token" pod "b457bfe9-1262-4ba5-bd19-fbdfd0e14310" (UID: "b457bfe9-1262-4ba5-bd19-fbdfd0e14310"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.895465    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.896563    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.896610    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.896625    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.896639    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16750225-7382-4170-8297-4def491a1ede-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.896654    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.896669    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16750225-7382-4170-8297-4def491a1ede-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.896681    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.896694    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.896707    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.896720    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16750225-7382-4170-8297-4def491a1ede-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.896733    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.896748    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.896763    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.896784    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.896817    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.896831    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5rck\" (UniqueName: \"kubernetes.io/projected/16750225-7382-4170-8297-4def491a1ede-kube-api-access-t5rck\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.896845    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16750225-7382-4170-8297-4def491a1ede-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:13.906847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.896862    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhjnw\" (UniqueName: \"kubernetes.io/projected/b457bfe9-1262-4ba5-bd19-fbdfd0e14310-kube-api-access-bhjnw\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:14.003822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.998483    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204e9634-b4e0-477d-8114-10fd7ecf0f72-kube-api-access-264f7\" (UniqueName: \"kubernetes.io/projected/204e9634-b4e0-477d-8114-10fd7ecf0f72-kube-api-access-264f7\") pod \"204e9634-b4e0-477d-8114-10fd7ecf0f72\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") "
Oct 29 23:03:14.003822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.998592    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-credential-socket\") pod \"204e9634-b4e0-477d-8114-10fd7ecf0f72\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") "
Oct 29 23:03:14.003822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.998697    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-podinfo\") pod \"204e9634-b4e0-477d-8114-10fd7ecf0f72\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") "
Oct 29 23:03:14.003822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.998748    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-data\") pod \"204e9634-b4e0-477d-8114-10fd7ecf0f72\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") "
Oct 29 23:03:14.003822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.998852    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-token\" (UniqueName: \"kubernetes.io/projected/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-token\") pod \"204e9634-b4e0-477d-8114-10fd7ecf0f72\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") "
Oct 29 23:03:14.003822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.998929    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-workload-socket\") pod \"204e9634-b4e0-477d-8114-10fd7ecf0f72\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") "
Oct 29 23:03:14.003822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.998993    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-envoy\") pod \"204e9634-b4e0-477d-8114-10fd7ecf0f72\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") "
Oct 29 23:03:14.003822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.999021    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-workload-certs\") pod \"204e9634-b4e0-477d-8114-10fd7ecf0f72\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") "
Oct 29 23:03:14.003822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:13.999128    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204e9634-b4e0-477d-8114-10fd7ecf0f72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204e9634-b4e0-477d-8114-10fd7ecf0f72-istiod-ca-cert\") pod \"204e9634-b4e0-477d-8114-10fd7ecf0f72\" (UID: \"204e9634-b4e0-477d-8114-10fd7ecf0f72\") "
Oct 29 23:03:14.019450 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.008897    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-data" pod "204e9634-b4e0-477d-8114-10fd7ecf0f72" (UID: "204e9634-b4e0-477d-8114-10fd7ecf0f72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:14.019450 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.018424    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-workload-certs" pod "204e9634-b4e0-477d-8114-10fd7ecf0f72" (UID: "204e9634-b4e0-477d-8114-10fd7ecf0f72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:14.019450 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.018839    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-workload-socket" pod "204e9634-b4e0-477d-8114-10fd7ecf0f72" (UID: "204e9634-b4e0-477d-8114-10fd7ecf0f72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:14.035777 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.023980    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204e9634-b4e0-477d-8114-10fd7ecf0f72-istiod-ca-cert" pod "204e9634-b4e0-477d-8114-10fd7ecf0f72" (UID: "204e9634-b4e0-477d-8114-10fd7ecf0f72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:14.035777 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.024066    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-credential-socket" pod "204e9634-b4e0-477d-8114-10fd7ecf0f72" (UID: "204e9634-b4e0-477d-8114-10fd7ecf0f72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:14.131456 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.104691    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204e9634-b4e0-477d-8114-10fd7ecf0f72-kube-api-access-264f7" pod "204e9634-b4e0-477d-8114-10fd7ecf0f72" (UID: "204e9634-b4e0-477d-8114-10fd7ecf0f72"). InnerVolumeSpecName "kube-api-access-264f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:14.131456 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.130341    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:14.131456 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.130373    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:14.131456 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.130386    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:14.131456 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.130399    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204e9634-b4e0-477d-8114-10fd7ecf0f72-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:14.131456 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.130411    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-264f7\" (UniqueName: \"kubernetes.io/projected/204e9634-b4e0-477d-8114-10fd7ecf0f72-kube-api-access-264f7\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:14.131456 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.130426    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:14.205142 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.205043    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-podinfo" pod "204e9634-b4e0-477d-8114-10fd7ecf0f72" (UID: "204e9634-b4e0-477d-8114-10fd7ecf0f72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:14.206088 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.205653    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-token" pod "204e9634-b4e0-477d-8114-10fd7ecf0f72" (UID: "204e9634-b4e0-477d-8114-10fd7ecf0f72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:14.207393 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.207367    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-envoy" pod "204e9634-b4e0-477d-8114-10fd7ecf0f72" (UID: "204e9634-b4e0-477d-8114-10fd7ecf0f72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:14.235900 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.235871    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:14.236098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.236083    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:14.236190 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.236179    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204e9634-b4e0-477d-8114-10fd7ecf0f72-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:14.390689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.390596    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"
Oct 29 23:03:14.391187 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.390703    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk" event={"ID":"16750225-7382-4170-8297-4def491a1ede","Type":"ContainerDied","Data":"4a7d9dffca273715b8794bb51e57cc7d9cbc8007764babecd58896277c67cc94"}
Oct 29 23:03:14.391187 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.390758    5428 scope.go:122] "RemoveContainer" containerID="bbccb5b86a5de6419444b08228a2af060bcc498019b5069073ac8151670bcb34"
Oct 29 23:03:14.397373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.397344    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2" event={"ID":"204e9634-b4e0-477d-8114-10fd7ecf0f72","Type":"ContainerDied","Data":"ac37a35e2d3e2b3b4e16dcec839d4caf836a3fd5306ad8192ea980b3f13c01a8"}
Oct 29 23:03:14.397617 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.397591    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"
Oct 29 23:03:14.401011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.400988    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm" event={"ID":"b457bfe9-1262-4ba5-bd19-fbdfd0e14310","Type":"ContainerDied","Data":"492124a193f64ba151284dba83eaf2be893c6789b943bfac4446e6d1a599e3fd"}
Oct 29 23:03:14.401123 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.401082    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"
Oct 29 23:03:14.418531 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.418506    5428 scope.go:122] "RemoveContainer" containerID="5b1e1e76ebe8ca74d49290ba27328a8c300cfcadcde8e9612f550e05cfb9abc9"
Oct 29 23:03:14.449949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.449923    5428 scope.go:122] "RemoveContainer" containerID="79aa1c26da7fd4420274106f637e173b8df7b354e5eb44e4389f34fe624abdb7"
Oct 29 23:03:14.590445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.590407    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"]
Oct 29 23:03:14.598668 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.598634    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-rcsr2"]
Oct 29 23:03:14.618353 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.618279    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"]
Oct 29 23:03:14.625494 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.625457    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-4r6rk"]
Oct 29 23:03:14.666369 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.666281    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"]
Oct 29 23:03:14.676871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.676839    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-fpvnm"]
Oct 29 23:03:14.787923 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.787883    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16750225-7382-4170-8297-4def491a1ede" path="/var/lib/kubelet/pods/16750225-7382-4170-8297-4def491a1ede/volumes"
Oct 29 23:03:14.788783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.788759    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204e9634-b4e0-477d-8114-10fd7ecf0f72" path="/var/lib/kubelet/pods/204e9634-b4e0-477d-8114-10fd7ecf0f72/volumes"
Oct 29 23:03:14.790218 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:14.790187    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b457bfe9-1262-4ba5-bd19-fbdfd0e14310" path="/var/lib/kubelet/pods/b457bfe9-1262-4ba5-bd19-fbdfd0e14310/volumes"
Oct 29 23:03:15.260988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:15.260953    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:16.232313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.232288    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 23:03:16.297465 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.297436    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 23:03:16.412231 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.412197    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-kube-api-access-p5mqq\" (UniqueName: \"kubernetes.io/projected/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-kube-api-access-p5mqq\") pod \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") "
Oct 29 23:03:16.412427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.412240    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istiod-ca-cert\") pod \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") "
Oct 29 23:03:16.412427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.412286    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-workload-certs\") pod \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") "
Oct 29 23:03:16.412427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.412364    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-data\") pod \"5b094a72-da28-4b91-9dec-f6d9d55543b9\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") "
Oct 29 23:03:16.412598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.412462    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-podinfo\") pod \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") "
Oct 29 23:03:16.412598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.412519    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-workload-socket\") pod \"5b094a72-da28-4b91-9dec-f6d9d55543b9\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") "
Oct 29 23:03:16.412598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.412556    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-data\") pod \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") "
Oct 29 23:03:16.412734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.412619    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-token\" (UniqueName: \"kubernetes.io/projected/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-token\") pod \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") "
Oct 29 23:03:16.412734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.412656    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-envoy\") pod \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") "
Oct 29 23:03:16.412734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.412706    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-workload-socket\") pod \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") "
Oct 29 23:03:16.412734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.412725    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istiod-ca-cert" pod "0855c417-df52-4f3d-8a49-bc3b1fb88a0f" (UID: "0855c417-df52-4f3d-8a49-bc3b1fb88a0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:16.413502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.412780    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-token\" (UniqueName: \"kubernetes.io/projected/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-token\") pod \"5b094a72-da28-4b91-9dec-f6d9d55543b9\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") "
Oct 29 23:03:16.413502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.412880    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-envoy\") pod \"5b094a72-da28-4b91-9dec-f6d9d55543b9\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") "
Oct 29 23:03:16.413502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.412955    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b094a72-da28-4b91-9dec-f6d9d55543b9-kube-api-access-9fwdv\" (UniqueName: \"kubernetes.io/projected/5b094a72-da28-4b91-9dec-f6d9d55543b9-kube-api-access-9fwdv\") pod \"5b094a72-da28-4b91-9dec-f6d9d55543b9\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") "
Oct 29 23:03:16.413502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.413009    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b094a72-da28-4b91-9dec-f6d9d55543b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b094a72-da28-4b91-9dec-f6d9d55543b9-istiod-ca-cert\") pod \"5b094a72-da28-4b91-9dec-f6d9d55543b9\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") "
Oct 29 23:03:16.413502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.413092    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-credential-socket\") pod \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\" (UID: \"0855c417-df52-4f3d-8a49-bc3b1fb88a0f\") "
Oct 29 23:03:16.413502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.413127    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-podinfo\") pod \"5b094a72-da28-4b91-9dec-f6d9d55543b9\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") "
Oct 29 23:03:16.413502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.413176    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-data" pod "5b094a72-da28-4b91-9dec-f6d9d55543b9" (UID: "5b094a72-da28-4b91-9dec-f6d9d55543b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:16.413502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.413203    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-credential-socket\") pod \"5b094a72-da28-4b91-9dec-f6d9d55543b9\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") "
Oct 29 23:03:16.413502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.413236    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-workload-certs\") pod \"5b094a72-da28-4b91-9dec-f6d9d55543b9\" (UID: \"5b094a72-da28-4b91-9dec-f6d9d55543b9\") "
Oct 29 23:03:16.413502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.413298    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-workload-socket" pod "5b094a72-da28-4b91-9dec-f6d9d55543b9" (UID: "5b094a72-da28-4b91-9dec-f6d9d55543b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:16.413502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.413435    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-data" pod "0855c417-df52-4f3d-8a49-bc3b1fb88a0f" (UID: "0855c417-df52-4f3d-8a49-bc3b1fb88a0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:16.414109 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.414019    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-workload-socket" pod "0855c417-df52-4f3d-8a49-bc3b1fb88a0f" (UID: "0855c417-df52-4f3d-8a49-bc3b1fb88a0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:16.414295 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.414268    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-workload-certs" pod "5b094a72-da28-4b91-9dec-f6d9d55543b9" (UID: "5b094a72-da28-4b91-9dec-f6d9d55543b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:16.414384 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.414359    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:16.414444 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.414390    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:16.414444 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.414404    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:16.414444 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.414418    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:16.414444 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.414430    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:16.415063 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.415030    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-podinfo" pod "0855c417-df52-4f3d-8a49-bc3b1fb88a0f" (UID: "0855c417-df52-4f3d-8a49-bc3b1fb88a0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:16.415322 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.415299    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-credential-socket" pod "0855c417-df52-4f3d-8a49-bc3b1fb88a0f" (UID: "0855c417-df52-4f3d-8a49-bc3b1fb88a0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:16.415442 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.415423    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-kube-api-access-p5mqq" pod "0855c417-df52-4f3d-8a49-bc3b1fb88a0f" (UID: "0855c417-df52-4f3d-8a49-bc3b1fb88a0f"). InnerVolumeSpecName "kube-api-access-p5mqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:16.415689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.415666    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-token" pod "0855c417-df52-4f3d-8a49-bc3b1fb88a0f" (UID: "0855c417-df52-4f3d-8a49-bc3b1fb88a0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:16.416202 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.416175    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-envoy" pod "5b094a72-da28-4b91-9dec-f6d9d55543b9" (UID: "5b094a72-da28-4b91-9dec-f6d9d55543b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:16.416202 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.416179    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-token" pod "5b094a72-da28-4b91-9dec-f6d9d55543b9" (UID: "5b094a72-da28-4b91-9dec-f6d9d55543b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:16.416919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.416892    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-credential-socket" pod "5b094a72-da28-4b91-9dec-f6d9d55543b9" (UID: "5b094a72-da28-4b91-9dec-f6d9d55543b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:16.416919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.416907    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-envoy" pod "0855c417-df52-4f3d-8a49-bc3b1fb88a0f" (UID: "0855c417-df52-4f3d-8a49-bc3b1fb88a0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:16.417362 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.417343    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b094a72-da28-4b91-9dec-f6d9d55543b9-kube-api-access-9fwdv" pod "5b094a72-da28-4b91-9dec-f6d9d55543b9" (UID: "5b094a72-da28-4b91-9dec-f6d9d55543b9"). InnerVolumeSpecName "kube-api-access-9fwdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:16.417362 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.417344    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-podinfo" pod "5b094a72-da28-4b91-9dec-f6d9d55543b9" (UID: "5b094a72-da28-4b91-9dec-f6d9d55543b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:16.424346 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.424320    5428 generic.go:356] "Generic (PLEG): container finished" podID="0855c417-df52-4f3d-8a49-bc3b1fb88a0f" containerID="dc0e9f6496bbd8ea4fd7921f803ad38eb4abd52b6bbab2c763eeaaeb55403585" exitCode=0
Oct 29 23:03:16.424426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.424366    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"
Oct 29 23:03:16.424426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.424400    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p" event={"ID":"0855c417-df52-4f3d-8a49-bc3b1fb88a0f","Type":"ContainerDied","Data":"dc0e9f6496bbd8ea4fd7921f803ad38eb4abd52b6bbab2c763eeaaeb55403585"}
Oct 29 23:03:16.424508 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.424453    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p" event={"ID":"0855c417-df52-4f3d-8a49-bc3b1fb88a0f","Type":"ContainerDied","Data":"15969f6c3f6e35e9a70f44794fde01534c703cb83665834127b745eb3709c8dc"}
Oct 29 23:03:16.424508 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.424471    5428 scope.go:122] "RemoveContainer" containerID="dc0e9f6496bbd8ea4fd7921f803ad38eb4abd52b6bbab2c763eeaaeb55403585"
Oct 29 23:03:16.426097 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.426075    5428 generic.go:356] "Generic (PLEG): container finished" podID="5b094a72-da28-4b91-9dec-f6d9d55543b9" containerID="661e425d0576452c701691e0015301a911143ab9a231b2b454f1609588c7df87" exitCode=0
Oct 29 23:03:16.426184 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.426125    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"
Oct 29 23:03:16.426262 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.426175    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f" event={"ID":"5b094a72-da28-4b91-9dec-f6d9d55543b9","Type":"ContainerDied","Data":"661e425d0576452c701691e0015301a911143ab9a231b2b454f1609588c7df87"}
Oct 29 23:03:16.426262 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.426229    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f" event={"ID":"5b094a72-da28-4b91-9dec-f6d9d55543b9","Type":"ContainerDied","Data":"c6f03efee1c167a63045c4bd3ef6a48e2ca7cfc940c39aeaf8dcaa115a3423c3"}
Oct 29 23:03:16.438467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.437061    5428 scope.go:122] "RemoveContainer" containerID="dc0e9f6496bbd8ea4fd7921f803ad38eb4abd52b6bbab2c763eeaaeb55403585"
Oct 29 23:03:16.438467 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:03:16.437392    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0e9f6496bbd8ea4fd7921f803ad38eb4abd52b6bbab2c763eeaaeb55403585\": container with ID starting with dc0e9f6496bbd8ea4fd7921f803ad38eb4abd52b6bbab2c763eeaaeb55403585 not found: ID does not exist" containerID="dc0e9f6496bbd8ea4fd7921f803ad38eb4abd52b6bbab2c763eeaaeb55403585"
Oct 29 23:03:16.438467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.437424    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0e9f6496bbd8ea4fd7921f803ad38eb4abd52b6bbab2c763eeaaeb55403585"} err="failed to get container status \"dc0e9f6496bbd8ea4fd7921f803ad38eb4abd52b6bbab2c763eeaaeb55403585\": rpc error: code = NotFound desc = could not find container \"dc0e9f6496bbd8ea4fd7921f803ad38eb4abd52b6bbab2c763eeaaeb55403585\": container with ID starting with dc0e9f6496bbd8ea4fd7921f803ad38eb4abd52b6bbab2c763eeaaeb55403585 not found: ID does not exist"
Oct 29 23:03:16.438467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.437442    5428 scope.go:122] "RemoveContainer" containerID="661e425d0576452c701691e0015301a911143ab9a231b2b454f1609588c7df87"
Oct 29 23:03:16.451503 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.450787    5428 scope.go:122] "RemoveContainer" containerID="661e425d0576452c701691e0015301a911143ab9a231b2b454f1609588c7df87"
Oct 29 23:03:16.451503 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:03:16.451154    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661e425d0576452c701691e0015301a911143ab9a231b2b454f1609588c7df87\": container with ID starting with 661e425d0576452c701691e0015301a911143ab9a231b2b454f1609588c7df87 not found: ID does not exist" containerID="661e425d0576452c701691e0015301a911143ab9a231b2b454f1609588c7df87"
Oct 29 23:03:16.451503 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.451179    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661e425d0576452c701691e0015301a911143ab9a231b2b454f1609588c7df87"} err="failed to get container status \"661e425d0576452c701691e0015301a911143ab9a231b2b454f1609588c7df87\": rpc error: code = NotFound desc = could not find container \"661e425d0576452c701691e0015301a911143ab9a231b2b454f1609588c7df87\": container with ID starting with 661e425d0576452c701691e0015301a911143ab9a231b2b454f1609588c7df87 not found: ID does not exist"
Oct 29 23:03:16.461561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.461532    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-workload-certs" pod "0855c417-df52-4f3d-8a49-bc3b1fb88a0f" (UID: "0855c417-df52-4f3d-8a49-bc3b1fb88a0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:16.461561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.461553    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b094a72-da28-4b91-9dec-f6d9d55543b9-istiod-ca-cert" pod "5b094a72-da28-4b91-9dec-f6d9d55543b9" (UID: "5b094a72-da28-4b91-9dec-f6d9d55543b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:16.515916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.515884    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:16.515916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.515912    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:16.515916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.515921    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5mqq\" (UniqueName: \"kubernetes.io/projected/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-kube-api-access-p5mqq\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:16.516143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.515931    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:16.516143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.515939    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:16.516143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.515947    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:16.516143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.515955    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:16.516143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.515962    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:16.516143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.515971    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:16.516143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.515978    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fwdv\" (UniqueName: \"kubernetes.io/projected/5b094a72-da28-4b91-9dec-f6d9d55543b9-kube-api-access-9fwdv\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:16.516143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.515986    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b094a72-da28-4b91-9dec-f6d9d55543b9-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:16.516143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.515994    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0855c417-df52-4f3d-8a49-bc3b1fb88a0f-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:16.516143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.516002    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b094a72-da28-4b91-9dec-f6d9d55543b9-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:16.779642 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.779613    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"]
Oct 29 23:03:16.791513 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.791481    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-vjp5f"]
Oct 29 23:03:16.816255 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.816208    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"]
Oct 29 23:03:16.821938 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.821913    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-kkx6p"]
Oct 29 23:03:16.961670 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.961635    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"]
Oct 29 23:03:16.961934 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.961898    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh" podUID="fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" containerName="istio-proxy" containerID="cri-o://9d19fcdd5fa352e69e566e0d23f5e2e66c30c5ef90ed5d7baa0604b99c47d43c" gracePeriod=30
Oct 29 23:03:16.968245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.968216    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"]
Oct 29 23:03:16.968421 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.968400    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh" podUID="751014a5-e82e-4403-83b7-f2e648514119" containerName="istio-proxy" containerID="cri-o://b166f7ea3195f011c8695a6d64b7dff6262d2c43cd621e6370964a3dbb90b78e" gracePeriod=30
Oct 29 23:03:16.972350 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.972320    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"]
Oct 29 23:03:16.972574 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.972548    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy" containerID="cri-o://4ffee01fbcaecb0deb64c6dc66834dd3d42d58dfd3da9b02bce5ebe092817eb7" gracePeriod=30
Oct 29 23:03:16.978756 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.978729    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"]
Oct 29 23:03:16.979009 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:16.978971    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy" containerID="cri-o://318bc9a014b14fe87878294d7866c39afd13e918b872df48056271089c144260" gracePeriod=30
Oct 29 23:03:17.336716 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:17.336675    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:17.391314 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:03:17.391276    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:03:18.602655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:18.602598    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:18.790194 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:18.790157    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:18.837693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:18.831583    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0855c417-df52-4f3d-8a49-bc3b1fb88a0f" path="/var/lib/kubelet/pods/0855c417-df52-4f3d-8a49-bc3b1fb88a0f/volumes"
Oct 29 23:03:18.837693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:18.832660    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b094a72-da28-4b91-9dec-f6d9d55543b9" path="/var/lib/kubelet/pods/5b094a72-da28-4b91-9dec-f6d9d55543b9/volumes"
Oct 29 23:03:19.349362 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:19.349211    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:03:19.349362 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:19.349329    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:03:19.390010 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:19.389880    5428 patch_prober.go:29] interesting pod/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:03:19.390261 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:19.390227    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:03:19.787610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:19.787582    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:20.554768 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:20.554729    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:20.834094 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:20.834019    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:20.937675 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:20.937645    5428 kubelet.go:1689] "Image garbage collection succeeded"
Oct 29 23:03:20.975859 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:20.975819    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:21.301245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:21.301211    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:22.311229 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.311197    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 23:03:22.358221 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.358187    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-data\") pod \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") "
Oct 29 23:03:22.358463 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.358275    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-kube-api-access-wklkj\" (UniqueName: \"kubernetes.io/projected/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-kube-api-access-wklkj\") pod \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") "
Oct 29 23:03:22.358463 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.358343    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-podinfo\") pod \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") "
Oct 29 23:03:22.358463 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.358414    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istiod-ca-cert\") pod \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") "
Oct 29 23:03:22.358463 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.358439    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-workload-socket\") pod \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") "
Oct 29 23:03:22.358716 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.358498    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-credential-socket\") pod \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") "
Oct 29 23:03:22.358716 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.358586    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-token\" (UniqueName: \"kubernetes.io/projected/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-token\") pod \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") "
Oct 29 23:03:22.358716 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.358706    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-workload-certs\") pod \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") "
Oct 29 23:03:22.358986 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.358963    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-envoy\") pod \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\" (UID: \"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4\") "
Oct 29 23:03:22.359564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.359535    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-data" pod "fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" (UID: "fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.359858 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.359796    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istiod-ca-cert" pod "fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" (UID: "fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:22.360140 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.360116    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-workload-certs" pod "fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" (UID: "fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.360225 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.360203    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-workload-socket" pod "fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" (UID: "fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.360297 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.360265    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-credential-socket" pod "fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" (UID: "fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.363020 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.362995    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-token" pod "fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" (UID: "fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:22.364389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.364363    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-kube-api-access-wklkj" pod "fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" (UID: "fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4"). InnerVolumeSpecName "kube-api-access-wklkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:22.364920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.364891    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-podinfo" pod "fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" (UID: "fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:22.364992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.364899    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-envoy" pod "fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" (UID: "fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.405519 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.405490    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 23:03:22.411033 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.411005    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 23:03:22.415820 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.415789    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 23:03:22.460693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.460664    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/751014a5-e82e-4403-83b7-f2e648514119-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/751014a5-e82e-4403-83b7-f2e648514119-istio-podinfo\") pod \"751014a5-e82e-4403-83b7-f2e648514119\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") "
Oct 29 23:03:22.460873 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.460700    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-token\" (UniqueName: \"kubernetes.io/projected/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-token\") pod \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") "
Oct 29 23:03:22.460873 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.460724    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/751014a5-e82e-4403-83b7-f2e648514119-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/751014a5-e82e-4403-83b7-f2e648514119-istiod-ca-cert\") pod \"751014a5-e82e-4403-83b7-f2e648514119\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") "
Oct 29 23:03:22.460873 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.460762    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-istio-envoy\") pod \"b7007018-33a5-4c01-b518-5a9f17273efb\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") "
Oct 29 23:03:22.460873 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.460824    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-workload-socket\") pod \"b7007018-33a5-4c01-b518-5a9f17273efb\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") "
Oct 29 23:03:22.460873 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.460861    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-data\") pod \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") "
Oct 29 23:03:22.461197 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.460889    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7007018-33a5-4c01-b518-5a9f17273efb-istio-token\" (UniqueName: \"kubernetes.io/projected/b7007018-33a5-4c01-b518-5a9f17273efb-istio-token\") pod \"b7007018-33a5-4c01-b518-5a9f17273efb\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") "
Oct 29 23:03:22.461197 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.460912    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-workload-certs\") pod \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") "
Oct 29 23:03:22.461197 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.460971    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istiod-ca-cert\") pod \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") "
Oct 29 23:03:22.461197 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461011    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-workload-certs\") pod \"b7007018-33a5-4c01-b518-5a9f17273efb\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") "
Oct 29 23:03:22.461197 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461049    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-istio-data\") pod \"751014a5-e82e-4403-83b7-f2e648514119\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") "
Oct 29 23:03:22.461197 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461195    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-istio-data\") pod \"b7007018-33a5-4c01-b518-5a9f17273efb\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") "
Oct 29 23:03:22.461489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461235    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-data" pod "91ad351f-5ccd-4777-8abd-7a02bdcb23eb" (UID: "91ad351f-5ccd-4777-8abd-7a02bdcb23eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.461489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461276    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-workload-socket\") pod \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") "
Oct 29 23:03:22.461489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461401    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-podinfo\") pod \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") "
Oct 29 23:03:22.461489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461455    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-credential-socket\") pod \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") "
Oct 29 23:03:22.461694 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461493    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-credential-socket\") pod \"751014a5-e82e-4403-83b7-f2e648514119\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") "
Oct 29 23:03:22.461694 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461551    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7007018-33a5-4c01-b518-5a9f17273efb-kube-api-access-crz95\" (UniqueName: \"kubernetes.io/projected/b7007018-33a5-4c01-b518-5a9f17273efb-kube-api-access-crz95\") pod \"b7007018-33a5-4c01-b518-5a9f17273efb\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") "
Oct 29 23:03:22.461694 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461595    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-workload-socket\") pod \"751014a5-e82e-4403-83b7-f2e648514119\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") "
Oct 29 23:03:22.461694 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461621    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-workload-socket" pod "91ad351f-5ccd-4777-8abd-7a02bdcb23eb" (UID: "91ad351f-5ccd-4777-8abd-7a02bdcb23eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.461694 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461642    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7007018-33a5-4c01-b518-5a9f17273efb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7007018-33a5-4c01-b518-5a9f17273efb-istiod-ca-cert\") pod \"b7007018-33a5-4c01-b518-5a9f17273efb\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") "
Oct 29 23:03:22.461973 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461700    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7007018-33a5-4c01-b518-5a9f17273efb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7007018-33a5-4c01-b518-5a9f17273efb-istio-podinfo\") pod \"b7007018-33a5-4c01-b518-5a9f17273efb\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") "
Oct 29 23:03:22.461973 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461764    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/751014a5-e82e-4403-83b7-f2e648514119-kube-api-access-6bn7v\" (UniqueName: \"kubernetes.io/projected/751014a5-e82e-4403-83b7-f2e648514119-kube-api-access-6bn7v\") pod \"751014a5-e82e-4403-83b7-f2e648514119\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") "
Oct 29 23:03:22.461973 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461844    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-workload-certs" pod "b7007018-33a5-4c01-b518-5a9f17273efb" (UID: "b7007018-33a5-4c01-b518-5a9f17273efb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.461973 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461869    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-envoy\") pod \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") "
Oct 29 23:03:22.461973 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461903    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-workload-certs\") pod \"751014a5-e82e-4403-83b7-f2e648514119\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") "
Oct 29 23:03:22.461973 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461951    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/751014a5-e82e-4403-83b7-f2e648514119-istio-token\" (UniqueName: \"kubernetes.io/projected/751014a5-e82e-4403-83b7-f2e648514119-istio-token\") pod \"751014a5-e82e-4403-83b7-f2e648514119\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") "
Oct 29 23:03:22.461973 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.461952    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istiod-ca-cert" pod "91ad351f-5ccd-4777-8abd-7a02bdcb23eb" (UID: "91ad351f-5ccd-4777-8abd-7a02bdcb23eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:22.462316 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.462019    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-credential-socket\") pod \"b7007018-33a5-4c01-b518-5a9f17273efb\" (UID: \"b7007018-33a5-4c01-b518-5a9f17273efb\") "
Oct 29 23:03:22.462316 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.462123    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-istio-envoy\") pod \"751014a5-e82e-4403-83b7-f2e648514119\" (UID: \"751014a5-e82e-4403-83b7-f2e648514119\") "
Oct 29 23:03:22.462316 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.462167    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-kube-api-access-plkrv\" (UniqueName: \"kubernetes.io/projected/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-kube-api-access-plkrv\") pod \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\" (UID: \"91ad351f-5ccd-4777-8abd-7a02bdcb23eb\") "
Oct 29 23:03:22.462316 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.462225    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-workload-certs" pod "91ad351f-5ccd-4777-8abd-7a02bdcb23eb" (UID: "91ad351f-5ccd-4777-8abd-7a02bdcb23eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.463064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.463032    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-istio-data" pod "751014a5-e82e-4403-83b7-f2e648514119" (UID: "751014a5-e82e-4403-83b7-f2e648514119"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.463161 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.463086    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751014a5-e82e-4403-83b7-f2e648514119-istiod-ca-cert" pod "751014a5-e82e-4403-83b7-f2e648514119" (UID: "751014a5-e82e-4403-83b7-f2e648514119"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:22.463283 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.463261    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-istio-envoy" pod "b7007018-33a5-4c01-b518-5a9f17273efb" (UID: "b7007018-33a5-4c01-b518-5a9f17273efb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.463346 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.463293    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-istio-data" pod "b7007018-33a5-4c01-b518-5a9f17273efb" (UID: "b7007018-33a5-4c01-b518-5a9f17273efb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.464415 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.464381    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/751014a5-e82e-4403-83b7-f2e648514119-istio-podinfo" pod "751014a5-e82e-4403-83b7-f2e648514119" (UID: "751014a5-e82e-4403-83b7-f2e648514119"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:22.464696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.464670    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-workload-socket" pod "b7007018-33a5-4c01-b518-5a9f17273efb" (UID: "b7007018-33a5-4c01-b518-5a9f17273efb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.464839 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.464707    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-credential-socket" pod "b7007018-33a5-4c01-b518-5a9f17273efb" (UID: "b7007018-33a5-4c01-b518-5a9f17273efb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.464839 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.464750    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7007018-33a5-4c01-b518-5a9f17273efb-istio-token" pod "b7007018-33a5-4c01-b518-5a9f17273efb" (UID: "b7007018-33a5-4c01-b518-5a9f17273efb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:22.465289 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.465260    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-workload-certs" pod "751014a5-e82e-4403-83b7-f2e648514119" (UID: "751014a5-e82e-4403-83b7-f2e648514119"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.465504 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.465482    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751014a5-e82e-4403-83b7-f2e648514119-kube-api-access-6bn7v" pod "751014a5-e82e-4403-83b7-f2e648514119" (UID: "751014a5-e82e-4403-83b7-f2e648514119"). InnerVolumeSpecName "kube-api-access-6bn7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:22.465582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.465558    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.465647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.465581    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.465647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.465596    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.465647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.465609    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.465647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.465623    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.465647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.465636    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.467609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.465649    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.467609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.465663    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wklkj\" (UniqueName: \"kubernetes.io/projected/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-kube-api-access-wklkj\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.467609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.465680    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.467609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.465694    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.467609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.465709    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.467609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.465722    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.467609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.465736    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.467609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.465750    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.467609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.465895    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-credential-socket" pod "751014a5-e82e-4403-83b7-f2e648514119" (UID: "751014a5-e82e-4403-83b7-f2e648514119"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.467609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.466311    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7007018-33a5-4c01-b518-5a9f17273efb-istio-podinfo" pod "b7007018-33a5-4c01-b518-5a9f17273efb" (UID: "b7007018-33a5-4c01-b518-5a9f17273efb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:22.467609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.466499    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-credential-socket" pod "91ad351f-5ccd-4777-8abd-7a02bdcb23eb" (UID: "91ad351f-5ccd-4777-8abd-7a02bdcb23eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.467609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.467492    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-workload-socket" pod "751014a5-e82e-4403-83b7-f2e648514119" (UID: "751014a5-e82e-4403-83b7-f2e648514119"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.468386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.467709    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-kube-api-access-plkrv" pod "91ad351f-5ccd-4777-8abd-7a02bdcb23eb" (UID: "91ad351f-5ccd-4777-8abd-7a02bdcb23eb"). InnerVolumeSpecName "kube-api-access-plkrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:22.468386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.467761    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7007018-33a5-4c01-b518-5a9f17273efb-istiod-ca-cert" pod "b7007018-33a5-4c01-b518-5a9f17273efb" (UID: "b7007018-33a5-4c01-b518-5a9f17273efb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:22.468882 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.468845    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-envoy" pod "91ad351f-5ccd-4777-8abd-7a02bdcb23eb" (UID: "91ad351f-5ccd-4777-8abd-7a02bdcb23eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.469416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.469395    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-podinfo" pod "91ad351f-5ccd-4777-8abd-7a02bdcb23eb" (UID: "91ad351f-5ccd-4777-8abd-7a02bdcb23eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:22.469416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.469398    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751014a5-e82e-4403-83b7-f2e648514119-istio-token" pod "751014a5-e82e-4403-83b7-f2e648514119" (UID: "751014a5-e82e-4403-83b7-f2e648514119"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:22.470128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.470101    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-istio-envoy" pod "751014a5-e82e-4403-83b7-f2e648514119" (UID: "751014a5-e82e-4403-83b7-f2e648514119"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:22.470483 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.470466    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7007018-33a5-4c01-b518-5a9f17273efb-kube-api-access-crz95" pod "b7007018-33a5-4c01-b518-5a9f17273efb" (UID: "b7007018-33a5-4c01-b518-5a9f17273efb"). InnerVolumeSpecName "kube-api-access-crz95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:22.496355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.496321    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-token" pod "91ad351f-5ccd-4777-8abd-7a02bdcb23eb" (UID: "91ad351f-5ccd-4777-8abd-7a02bdcb23eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:22.521367 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.521287    5428 generic.go:356] "Generic (PLEG): container finished" podID="fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" containerID="9d19fcdd5fa352e69e566e0d23f5e2e66c30c5ef90ed5d7baa0604b99c47d43c" exitCode=0
Oct 29 23:03:22.521367 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.521350    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"
Oct 29 23:03:22.521576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.521364    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh" event={"ID":"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4","Type":"ContainerDied","Data":"9d19fcdd5fa352e69e566e0d23f5e2e66c30c5ef90ed5d7baa0604b99c47d43c"}
Oct 29 23:03:22.521576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.521458    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh" event={"ID":"fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4","Type":"ContainerDied","Data":"1456a15fef9c59d3564eca75aefe1f8eee4c75ced9d7cf1b603109cd4f5fa58e"}
Oct 29 23:03:22.521576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.521493    5428 scope.go:122] "RemoveContainer" containerID="9d19fcdd5fa352e69e566e0d23f5e2e66c30c5ef90ed5d7baa0604b99c47d43c"
Oct 29 23:03:22.523194 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.523171    5428 generic.go:356] "Generic (PLEG): container finished" podID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerID="4ffee01fbcaecb0deb64c6dc66834dd3d42d58dfd3da9b02bce5ebe092817eb7" exitCode=0
Oct 29 23:03:22.523882 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.523226    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"
Oct 29 23:03:22.523882 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.523245    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" event={"ID":"91ad351f-5ccd-4777-8abd-7a02bdcb23eb","Type":"ContainerDied","Data":"4ffee01fbcaecb0deb64c6dc66834dd3d42d58dfd3da9b02bce5ebe092817eb7"}
Oct 29 23:03:22.523882 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.523526    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794" event={"ID":"91ad351f-5ccd-4777-8abd-7a02bdcb23eb","Type":"ContainerDied","Data":"26f983fc376a6e45e6f4f15fa7141e4cfc6ee12846ebb3ce3d8aa2764dfbea45"}
Oct 29 23:03:22.527620 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.526442    5428 generic.go:356] "Generic (PLEG): container finished" podID="751014a5-e82e-4403-83b7-f2e648514119" containerID="b166f7ea3195f011c8695a6d64b7dff6262d2c43cd621e6370964a3dbb90b78e" exitCode=0
Oct 29 23:03:22.527620 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.526493    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"
Oct 29 23:03:22.527620 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.526491    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh" event={"ID":"751014a5-e82e-4403-83b7-f2e648514119","Type":"ContainerDied","Data":"b166f7ea3195f011c8695a6d64b7dff6262d2c43cd621e6370964a3dbb90b78e"}
Oct 29 23:03:22.527620 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.526921    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh" event={"ID":"751014a5-e82e-4403-83b7-f2e648514119","Type":"ContainerDied","Data":"2fc54d46dbcc7d0362917f24b18c3c902fb165e718df269c19fb5e76ec6e45bb"}
Oct 29 23:03:22.528357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.528330    5428 generic.go:356] "Generic (PLEG): container finished" podID="b7007018-33a5-4c01-b518-5a9f17273efb" containerID="318bc9a014b14fe87878294d7866c39afd13e918b872df48056271089c144260" exitCode=0
Oct 29 23:03:22.528491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.528472    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"
Oct 29 23:03:22.528587 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.528568    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" event={"ID":"b7007018-33a5-4c01-b518-5a9f17273efb","Type":"ContainerDied","Data":"318bc9a014b14fe87878294d7866c39afd13e918b872df48056271089c144260"}
Oct 29 23:03:22.528654 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.528616    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl" event={"ID":"b7007018-33a5-4c01-b518-5a9f17273efb","Type":"ContainerDied","Data":"ee025b793e3011c98fb1abae3b3e05cfaadb6788c28fdce75f32c5b8f1ed9d59"}
Oct 29 23:03:22.540299 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.540278    5428 scope.go:122] "RemoveContainer" containerID="9d19fcdd5fa352e69e566e0d23f5e2e66c30c5ef90ed5d7baa0604b99c47d43c"
Oct 29 23:03:22.540670 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:03:22.540636    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d19fcdd5fa352e69e566e0d23f5e2e66c30c5ef90ed5d7baa0604b99c47d43c\": container with ID starting with 9d19fcdd5fa352e69e566e0d23f5e2e66c30c5ef90ed5d7baa0604b99c47d43c not found: ID does not exist" containerID="9d19fcdd5fa352e69e566e0d23f5e2e66c30c5ef90ed5d7baa0604b99c47d43c"
Oct 29 23:03:22.540754 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.540667    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d19fcdd5fa352e69e566e0d23f5e2e66c30c5ef90ed5d7baa0604b99c47d43c"} err="failed to get container status \"9d19fcdd5fa352e69e566e0d23f5e2e66c30c5ef90ed5d7baa0604b99c47d43c\": rpc error: code = NotFound desc = could not find container \"9d19fcdd5fa352e69e566e0d23f5e2e66c30c5ef90ed5d7baa0604b99c47d43c\": container with ID starting with 9d19fcdd5fa352e69e566e0d23f5e2e66c30c5ef90ed5d7baa0604b99c47d43c not found: ID does not exist"
Oct 29 23:03:22.540754 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.540692    5428 scope.go:122] "RemoveContainer" containerID="4ffee01fbcaecb0deb64c6dc66834dd3d42d58dfd3da9b02bce5ebe092817eb7"
Oct 29 23:03:22.559178 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.559151    5428 scope.go:122] "RemoveContainer" containerID="4ffee01fbcaecb0deb64c6dc66834dd3d42d58dfd3da9b02bce5ebe092817eb7"
Oct 29 23:03:22.559585 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:03:22.559555    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ffee01fbcaecb0deb64c6dc66834dd3d42d58dfd3da9b02bce5ebe092817eb7\": container with ID starting with 4ffee01fbcaecb0deb64c6dc66834dd3d42d58dfd3da9b02bce5ebe092817eb7 not found: ID does not exist" containerID="4ffee01fbcaecb0deb64c6dc66834dd3d42d58dfd3da9b02bce5ebe092817eb7"
Oct 29 23:03:22.559693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.559596    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ffee01fbcaecb0deb64c6dc66834dd3d42d58dfd3da9b02bce5ebe092817eb7"} err="failed to get container status \"4ffee01fbcaecb0deb64c6dc66834dd3d42d58dfd3da9b02bce5ebe092817eb7\": rpc error: code = NotFound desc = could not find container \"4ffee01fbcaecb0deb64c6dc66834dd3d42d58dfd3da9b02bce5ebe092817eb7\": container with ID starting with 4ffee01fbcaecb0deb64c6dc66834dd3d42d58dfd3da9b02bce5ebe092817eb7 not found: ID does not exist"
Oct 29 23:03:22.559693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.559624    5428 scope.go:122] "RemoveContainer" containerID="b166f7ea3195f011c8695a6d64b7dff6262d2c43cd621e6370964a3dbb90b78e"
Oct 29 23:03:22.567669 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567645    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7007018-33a5-4c01-b518-5a9f17273efb-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.567669 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567670    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.567810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567679    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.567810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567687    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.567810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567696    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.567810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567709    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.567810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567719    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crz95\" (UniqueName: \"kubernetes.io/projected/b7007018-33a5-4c01-b518-5a9f17273efb-kube-api-access-crz95\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.567810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567727    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.567810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567736    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7007018-33a5-4c01-b518-5a9f17273efb-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.567810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567744    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7007018-33a5-4c01-b518-5a9f17273efb-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.567810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567751    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bn7v\" (UniqueName: \"kubernetes.io/projected/751014a5-e82e-4403-83b7-f2e648514119-kube-api-access-6bn7v\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.567810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567760    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.567810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567767    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.567810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567776    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/751014a5-e82e-4403-83b7-f2e648514119-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.567810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567783    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.567810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567791    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/751014a5-e82e-4403-83b7-f2e648514119-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.567810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567815    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plkrv\" (UniqueName: \"kubernetes.io/projected/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-kube-api-access-plkrv\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.569132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567825    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/751014a5-e82e-4403-83b7-f2e648514119-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.569132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567834    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ad351f-5ccd-4777-8abd-7a02bdcb23eb-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.569132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567842    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/751014a5-e82e-4403-83b7-f2e648514119-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.569132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567849    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.569132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.567857    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7007018-33a5-4c01-b518-5a9f17273efb-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.582551 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.582534    5428 scope.go:122] "RemoveContainer" containerID="80c7dcab7d14ea41d1fe5c5dcb6307b187bd8377b64b6d99b3c72e48e4a1b7d9"
Oct 29 23:03:22.601712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.601685    5428 scope.go:122] "RemoveContainer" containerID="b166f7ea3195f011c8695a6d64b7dff6262d2c43cd621e6370964a3dbb90b78e"
Oct 29 23:03:22.602164 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:03:22.602141    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b166f7ea3195f011c8695a6d64b7dff6262d2c43cd621e6370964a3dbb90b78e\": container with ID starting with b166f7ea3195f011c8695a6d64b7dff6262d2c43cd621e6370964a3dbb90b78e not found: ID does not exist" containerID="b166f7ea3195f011c8695a6d64b7dff6262d2c43cd621e6370964a3dbb90b78e"
Oct 29 23:03:22.602286 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.602173    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b166f7ea3195f011c8695a6d64b7dff6262d2c43cd621e6370964a3dbb90b78e"} err="failed to get container status \"b166f7ea3195f011c8695a6d64b7dff6262d2c43cd621e6370964a3dbb90b78e\": rpc error: code = NotFound desc = could not find container \"b166f7ea3195f011c8695a6d64b7dff6262d2c43cd621e6370964a3dbb90b78e\": container with ID starting with b166f7ea3195f011c8695a6d64b7dff6262d2c43cd621e6370964a3dbb90b78e not found: ID does not exist"
Oct 29 23:03:22.602286 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.602192    5428 scope.go:122] "RemoveContainer" containerID="80c7dcab7d14ea41d1fe5c5dcb6307b187bd8377b64b6d99b3c72e48e4a1b7d9"
Oct 29 23:03:22.603243 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:03:22.603213    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80c7dcab7d14ea41d1fe5c5dcb6307b187bd8377b64b6d99b3c72e48e4a1b7d9\": container with ID starting with 80c7dcab7d14ea41d1fe5c5dcb6307b187bd8377b64b6d99b3c72e48e4a1b7d9 not found: ID does not exist" containerID="80c7dcab7d14ea41d1fe5c5dcb6307b187bd8377b64b6d99b3c72e48e4a1b7d9"
Oct 29 23:03:22.603380 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.603353    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c7dcab7d14ea41d1fe5c5dcb6307b187bd8377b64b6d99b3c72e48e4a1b7d9"} err="failed to get container status \"80c7dcab7d14ea41d1fe5c5dcb6307b187bd8377b64b6d99b3c72e48e4a1b7d9\": rpc error: code = NotFound desc = could not find container \"80c7dcab7d14ea41d1fe5c5dcb6307b187bd8377b64b6d99b3c72e48e4a1b7d9\": container with ID starting with 80c7dcab7d14ea41d1fe5c5dcb6307b187bd8377b64b6d99b3c72e48e4a1b7d9 not found: ID does not exist"
Oct 29 23:03:22.603380 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.603380    5428 scope.go:122] "RemoveContainer" containerID="318bc9a014b14fe87878294d7866c39afd13e918b872df48056271089c144260"
Oct 29 23:03:22.610797 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.610774    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"]
Oct 29 23:03:22.620976 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.620950    5428 scope.go:122] "RemoveContainer" containerID="318bc9a014b14fe87878294d7866c39afd13e918b872df48056271089c144260"
Oct 29 23:03:22.621321 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:03:22.621296    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"318bc9a014b14fe87878294d7866c39afd13e918b872df48056271089c144260\": container with ID starting with 318bc9a014b14fe87878294d7866c39afd13e918b872df48056271089c144260 not found: ID does not exist" containerID="318bc9a014b14fe87878294d7866c39afd13e918b872df48056271089c144260"
Oct 29 23:03:22.621426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.621328    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"318bc9a014b14fe87878294d7866c39afd13e918b872df48056271089c144260"} err="failed to get container status \"318bc9a014b14fe87878294d7866c39afd13e918b872df48056271089c144260\": rpc error: code = NotFound desc = could not find container \"318bc9a014b14fe87878294d7866c39afd13e918b872df48056271089c144260\": container with ID starting with 318bc9a014b14fe87878294d7866c39afd13e918b872df48056271089c144260 not found: ID does not exist"
Oct 29 23:03:22.624034 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.624011    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-pzmnh"]
Oct 29 23:03:22.643755 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.643727    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"]
Oct 29 23:03:22.649708 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.649682    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hk794"]
Oct 29 23:03:22.685068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.685031    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"]
Oct 29 23:03:22.693949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.693910    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-hrfzh"]
Oct 29 23:03:22.729461 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.729423    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"]
Oct 29 23:03:22.736390 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.736356    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-32c192-openshift-default-8dbd47ddc-nxzxl"]
Oct 29 23:03:22.781302 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.781227    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="751014a5-e82e-4403-83b7-f2e648514119" path="/var/lib/kubelet/pods/751014a5-e82e-4403-83b7-f2e648514119/volumes"
Oct 29 23:03:22.782482 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.782457    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" path="/var/lib/kubelet/pods/91ad351f-5ccd-4777-8abd-7a02bdcb23eb/volumes"
Oct 29 23:03:22.783474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.783452    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" path="/var/lib/kubelet/pods/b7007018-33a5-4c01-b518-5a9f17273efb/volumes"
Oct 29 23:03:22.784326 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:22.784309    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" path="/var/lib/kubelet/pods/fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4/volumes"
Oct 29 23:03:23.311012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:23.310975    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:25.112201 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.112165    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:25.228752 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.228718    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:25.767294 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.767260    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"]
Oct 29 23:03:25.768032 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768016    5428 cpu_manager.go:414] "containerMap: removing container" podUID="fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" containerName="istio-proxy"
Oct 29 23:03:25.768081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768034    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" containerName="istio-proxy"
Oct 29 23:03:25.768081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768051    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b457bfe9-1262-4ba5-bd19-fbdfd0e14310" containerName="istio-proxy"
Oct 29 23:03:25.768081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768056    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b457bfe9-1262-4ba5-bd19-fbdfd0e14310" containerName="istio-proxy"
Oct 29 23:03:25.768081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768066    5428 cpu_manager.go:414] "containerMap: removing container" podUID="5b094a72-da28-4b91-9dec-f6d9d55543b9" containerName="istio-proxy"
Oct 29 23:03:25.768081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768071    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b094a72-da28-4b91-9dec-f6d9d55543b9" containerName="istio-proxy"
Oct 29 23:03:25.768242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768087    5428 cpu_manager.go:414] "containerMap: removing container" podUID="16750225-7382-4170-8297-4def491a1ede" containerName="istio-proxy"
Oct 29 23:03:25.768242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768092    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16750225-7382-4170-8297-4def491a1ede" containerName="istio-proxy"
Oct 29 23:03:25.768242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768102    5428 cpu_manager.go:414] "containerMap: removing container" podUID="751014a5-e82e-4403-83b7-f2e648514119" containerName="istio-proxy"
Oct 29 23:03:25.768242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768106    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="751014a5-e82e-4403-83b7-f2e648514119" containerName="istio-proxy"
Oct 29 23:03:25.768242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768112    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy"
Oct 29 23:03:25.768242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768117    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy"
Oct 29 23:03:25.768242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768130    5428 cpu_manager.go:414] "containerMap: removing container" podUID="751014a5-e82e-4403-83b7-f2e648514119" containerName="istio-proxy"
Oct 29 23:03:25.768242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768135    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="751014a5-e82e-4403-83b7-f2e648514119" containerName="istio-proxy"
Oct 29 23:03:25.768242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768143    5428 cpu_manager.go:414] "containerMap: removing container" podUID="204e9634-b4e0-477d-8114-10fd7ecf0f72" containerName="istio-proxy"
Oct 29 23:03:25.768242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768147    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204e9634-b4e0-477d-8114-10fd7ecf0f72" containerName="istio-proxy"
Oct 29 23:03:25.768242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768154    5428 cpu_manager.go:414] "containerMap: removing container" podUID="a32d5c46-fd89-4d2e-a902-9c3c80abfcb7" containerName="payload-pre-processing"
Oct 29 23:03:25.768242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768158    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a32d5c46-fd89-4d2e-a902-9c3c80abfcb7" containerName="payload-pre-processing"
Oct 29 23:03:25.768242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768169    5428 cpu_manager.go:414] "containerMap: removing container" podUID="63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" containerName="main"
Oct 29 23:03:25.768242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768173    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" containerName="main"
Oct 29 23:03:25.768242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768185    5428 cpu_manager.go:414] "containerMap: removing container" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy"
Oct 29 23:03:25.768242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768189    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy"
Oct 29 23:03:25.768242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768204    5428 cpu_manager.go:414] "containerMap: removing container" podUID="0855c417-df52-4f3d-8a49-bc3b1fb88a0f" containerName="istio-proxy"
Oct 29 23:03:25.768242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768209    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0855c417-df52-4f3d-8a49-bc3b1fb88a0f" containerName="istio-proxy"
Oct 29 23:03:25.769492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768298    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ad351f-5ccd-4777-8abd-7a02bdcb23eb" containerName="istio-proxy"
Oct 29 23:03:25.769492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768310    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="0855c417-df52-4f3d-8a49-bc3b1fb88a0f" containerName="istio-proxy"
Oct 29 23:03:25.769492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768320    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcb3a728-4b70-4a8b-bdd3-6fe99b4fa3d4" containerName="istio-proxy"
Oct 29 23:03:25.769492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768326    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b457bfe9-1262-4ba5-bd19-fbdfd0e14310" containerName="istio-proxy"
Oct 29 23:03:25.769492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768333    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b094a72-da28-4b91-9dec-f6d9d55543b9" containerName="istio-proxy"
Oct 29 23:03:25.769492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768358    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="16750225-7382-4170-8297-4def491a1ede" containerName="istio-proxy"
Oct 29 23:03:25.769492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768364    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="751014a5-e82e-4403-83b7-f2e648514119" containerName="istio-proxy"
Oct 29 23:03:25.769492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768369    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7007018-33a5-4c01-b518-5a9f17273efb" containerName="istio-proxy"
Oct 29 23:03:25.769492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768376    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="204e9634-b4e0-477d-8114-10fd7ecf0f72" containerName="istio-proxy"
Oct 29 23:03:25.769492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768383    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a32d5c46-fd89-4d2e-a902-9c3c80abfcb7" containerName="payload-pre-processing"
Oct 29 23:03:25.769492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.768389    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d3a89f-0c81-4eeb-b0fa-8d3cd8d497db" containerName="main"
Oct 29 23:03:25.774861 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.774833    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.800819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.800599    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"]
Oct 29 23:03:25.801436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.801398    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-44a4e4-openshift-default-dockercfg-qjfbh\""
Oct 29 23:03:25.831505 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.831463    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-podinfo\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.831688 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.831629    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-data\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.831766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.831725    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-envoy\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.832221 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.832195    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-workload-certs\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.834464 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.833627    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhxbk\" (UniqueName: \"kubernetes.io/projected/cb016a04-d965-4ee7-b802-48c35f8c4b61-kube-api-access-vhxbk\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.834464 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.833728    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-workload-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.834464 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.833854    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-token\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.834464 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.834006    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb016a04-d965-4ee7-b802-48c35f8c4b61-istiod-ca-cert\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.834464 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.834078    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-credential-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.935476 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.935441    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-envoy\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.935707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.935504    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-workload-certs\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.935707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.935546    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhxbk\" (UniqueName: \"kubernetes.io/projected/cb016a04-d965-4ee7-b802-48c35f8c4b61-kube-api-access-vhxbk\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.935707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.935575    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-workload-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.935707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.935642    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-token\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.935991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.935743    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb016a04-d965-4ee7-b802-48c35f8c4b61-istiod-ca-cert\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.935991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.935793    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-credential-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.936263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.936235    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-workload-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.936564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.936535    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-workload-certs\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.936660 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.936538    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-podinfo\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.936761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.936711    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-credential-socket\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.936920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.936787    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-data\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.936920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.936912    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb016a04-d965-4ee7-b802-48c35f8c4b61-istiod-ca-cert\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.937204 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.937169    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-data\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.943634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.939670    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-podinfo\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.948761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.945187    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:25.948761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.945274    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-envoy\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.960500 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.960474    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-token\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:25.960662 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:25.960503    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhxbk\" (UniqueName: \"kubernetes.io/projected/cb016a04-d965-4ee7-b802-48c35f8c4b61-kube-api-access-vhxbk\") pod \"e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") " pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:26.107505 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:26.107422    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:26.380268 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:26.380230    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:26.439757 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:26.439723    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"]
Oct 29 23:03:26.439981 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:03:26.439957    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb016a04_d965_4ee7_b802_48c35f8c4b61.slice/crio-631ad04b3020e46925cae0dd756ded7c060ca34a2796871e1d30dbd75901238d WatchSource:0}: Error finding container 631ad04b3020e46925cae0dd756ded7c060ca34a2796871e1d30dbd75901238d: Status 404 returned error can't find the container with id 631ad04b3020e46925cae0dd756ded7c060ca34a2796871e1d30dbd75901238d
Oct 29 23:03:26.442686 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:26.442651    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:03:26.442783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:26.442718    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:03:26.442783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:26.442747    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:03:26.499557 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:26.499531    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:26.570509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:26.570472    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp" event={"ID":"cb016a04-d965-4ee7-b802-48c35f8c4b61","Type":"ContainerStarted","Data":"98f1b02998d9a24acb564d647d8a9e7403693d102e113562dad09020bb765cfc"}
Oct 29 23:03:26.570706 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:26.570531    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp" event={"ID":"cb016a04-d965-4ee7-b802-48c35f8c4b61","Type":"ContainerStarted","Data":"631ad04b3020e46925cae0dd756ded7c060ca34a2796871e1d30dbd75901238d"}
Oct 29 23:03:26.615339 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:26.615274    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp" podStartSLOduration=1.615255453 podStartE2EDuration="1.615255453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:26.614053061 +0000 UTC m=+1209.957001818" watchObservedRunningTime="2026-10-29 23:03:26.615255453 +0000 UTC m=+1209.958204199"
Oct 29 23:03:27.107728 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:27.107695    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:27.109904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:27.109874    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 29 23:03:27.110719 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:27.110073    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp" podUID="cb016a04-d965-4ee7-b802-48c35f8c4b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 29 23:03:27.814995 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:27.814962    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:27.989697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:27.989657    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:28.109157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:28.108634    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 29 23:03:28.109157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:28.108695    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp" podUID="cb016a04-d965-4ee7-b802-48c35f8c4b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 29 23:03:28.293900 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:28.293856    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:28.458729 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:28.458695    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:28.835925 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:28.828713    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:29.108231 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:29.108150    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 29 23:03:29.108231 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:29.108212    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp" podUID="cb016a04-d965-4ee7-b802-48c35f8c4b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 29 23:03:29.659978 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:29.659943    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:30.109269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:30.109189    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 29 23:03:30.109624 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:30.109259    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp" podUID="cb016a04-d965-4ee7-b802-48c35f8c4b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 29 23:03:30.204689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:30.204635    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:30.647419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:30.647376    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:31.108717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:31.108639    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 29 23:03:31.108717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:31.108688    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp" podUID="cb016a04-d965-4ee7-b802-48c35f8c4b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 29 23:03:32.108308 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:32.108272    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 29 23:03:32.108772 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:32.108332    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp" podUID="cb016a04-d965-4ee7-b802-48c35f8c4b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 29 23:03:32.184845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:32.184757    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:32.455008 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:32.454958    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:32.605952 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:32.605917    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:32.826167 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:32.826077    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"]
Oct 29 23:03:32.826356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:32.826297    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll" podUID="b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" containerName="istio-proxy" containerID="cri-o://e45987552fa8976f9c466a7e8f0717443c9563e07e22dae211aced093b1260f3" gracePeriod=30
Oct 29 23:03:32.832665 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:32.832632    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:33.108766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:33.108679    5428 patch_prober.go:29] interesting pod/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 29 23:03:33.108766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:33.108747    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp" podUID="cb016a04-d965-4ee7-b802-48c35f8c4b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 29 23:03:33.642712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:33.642674    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:34.113261 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:34.113184    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:34.113840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:34.113692    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:34.114477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:34.114454    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:03:34.554342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:34.554301    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:35.418735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:35.418699    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:35.544465 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:35.544428    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:35.888424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:35.888388    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"]
Oct 29 23:03:35.888721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:35.888694    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx" podUID="86892f1f-cbce-48a4-80ae-48e4c0f88abf" containerName="istio-proxy" containerID="cri-o://f1fd7a3a2fab6e67ffda72157861716c9386953c97576bc67b32628f2b0a7ff5" gracePeriod=30
Oct 29 23:03:35.900296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:35.900266    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"]
Oct 29 23:03:35.900594 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:35.900571    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm" podUID="e415acf5-67c8-47c1-aa03-df58dca41b3a" containerName="istio-proxy" containerID="cri-o://6f4f37945e2bebf1405ffe03edddc744f8e15819e70efe15093d46c7b957cb08" gracePeriod=30
Oct 29 23:03:35.905497 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:35.905473    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"]
Oct 29 23:03:35.905711 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:35.905689    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z" podUID="b20a2142-6441-4846-8855-b4595d2fe746" containerName="istio-proxy" containerID="cri-o://79a1b80d99a5ad05884bf5ec2a2cfab7b78c1b200ff75974faad6081ff247e7f" gracePeriod=30
Oct 29 23:03:36.101002 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.100963    5428 patch_prober.go:29] interesting pod/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:03:36.101166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.101032    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll" podUID="b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:03:36.635955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.635922    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"]
Oct 29 23:03:36.636793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.636778    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="751014a5-e82e-4403-83b7-f2e648514119" containerName="istio-proxy"
Oct 29 23:03:36.640559 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.640529    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.676291 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.676253    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"]
Oct 29 23:03:36.741356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.741330    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-10b990-openshift-default-dockercfg-nnpxx\""
Oct 29 23:03:36.743224 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.743198    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.743332 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.743259    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.743375 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.743340    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btv2m\" (UniqueName: \"kubernetes.io/projected/b886f3b9-2571-4566-9c33-4fea56038612-kube-api-access-btv2m\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.743513 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.743491    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.743615 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.743602    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b886f3b9-2571-4566-9c33-4fea56038612-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.743687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.743672    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b886f3b9-2571-4566-9c33-4fea56038612-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.743725 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.743710    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.743784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.743773    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.743850 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.743823    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b886f3b9-2571-4566-9c33-4fea56038612-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.845542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.845507    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.845763 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.845682    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b886f3b9-2571-4566-9c33-4fea56038612-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.845763 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.845750    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b886f3b9-2571-4566-9c33-4fea56038612-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.845955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.845778    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.845955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.845866    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.845955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.845896    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b886f3b9-2571-4566-9c33-4fea56038612-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.845955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.845940    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.846157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.846002    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.846157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.846047    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btv2m\" (UniqueName: \"kubernetes.io/projected/b886f3b9-2571-4566-9c33-4fea56038612-kube-api-access-btv2m\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.846255 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.846223    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.846301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.846001    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.846584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.846552    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.846710 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.846590    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b886f3b9-2571-4566-9c33-4fea56038612-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.846710 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.846611    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.848207 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.848188    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.848450 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.848432    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b886f3b9-2571-4566-9c33-4fea56038612-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.868504 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.868477    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b886f3b9-2571-4566-9c33-4fea56038612-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.868679 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.868661    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btv2m\" (UniqueName: \"kubernetes.io/projected/b886f3b9-2571-4566-9c33-4fea56038612-kube-api-access-btv2m\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:36.956863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:36.956757    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:37.291442 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:37.291412    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"]
Oct 29 23:03:37.292690 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:03:37.292663    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb886f3b9_2571_4566_9c33_4fea56038612.slice/crio-07ad30eea6b3652573f1a38790ae3f66aa271718a0caec706fa40606b045a1d2 WatchSource:0}: Error finding container 07ad30eea6b3652573f1a38790ae3f66aa271718a0caec706fa40606b045a1d2: Status 404 returned error can't find the container with id 07ad30eea6b3652573f1a38790ae3f66aa271718a0caec706fa40606b045a1d2
Oct 29 23:03:37.295574 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:37.295518    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:03:37.295674 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:37.295625    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:03:37.295746 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:37.295672    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:03:37.708906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:37.703884    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8" event={"ID":"b886f3b9-2571-4566-9c33-4fea56038612","Type":"ContainerStarted","Data":"252f9e693c00d8c9c2d86ba19253598581bbfd856960d23fe3d9d4694d66b31e"}
Oct 29 23:03:37.708906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:37.703961    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8" event={"ID":"b886f3b9-2571-4566-9c33-4fea56038612","Type":"ContainerStarted","Data":"07ad30eea6b3652573f1a38790ae3f66aa271718a0caec706fa40606b045a1d2"}
Oct 29 23:03:37.955994 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:37.955966    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:37.957692 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:37.957671    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:38.070351 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.070284    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:38.593451 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.593420    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 23:03:38.658071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.658007    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8" podStartSLOduration=2.657987713 podStartE2EDuration="2.657987713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:37.755007691 +0000 UTC m=+1221.097956436" watchObservedRunningTime="2026-10-29 23:03:38.657987713 +0000 UTC m=+1222.000936459"
Oct 29 23:03:38.658717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.658695    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-envoy\") pod \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") "
Oct 29 23:03:38.658845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.658824    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-data\") pod \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") "
Oct 29 23:03:38.658978 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.658870    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istiod-ca-cert\") pod \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") "
Oct 29 23:03:38.659458 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.659090    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-workload-socket\") pod \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") "
Oct 29 23:03:38.659458 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.659167    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-kube-api-access-z274x\" (UniqueName: \"kubernetes.io/projected/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-kube-api-access-z274x\") pod \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") "
Oct 29 23:03:38.659936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.659911    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-workload-certs\") pod \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") "
Oct 29 23:03:38.660114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.660094    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-credential-socket\") pod \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") "
Oct 29 23:03:38.660233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.660218    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-token\" (UniqueName: \"kubernetes.io/projected/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-token\") pod \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") "
Oct 29 23:03:38.660353 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.660339    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-podinfo\") pod \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\" (UID: \"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff\") "
Oct 29 23:03:38.664941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.663979    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istiod-ca-cert" pod "b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" (UID: "b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:38.665354 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.665089    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-workload-socket" pod "b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" (UID: "b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:38.682353 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.667880    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-data" pod "b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" (UID: "b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:38.682353 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.668093    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-credential-socket" pod "b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" (UID: "b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:38.682353 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.671874    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-workload-certs" pod "b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" (UID: "b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:38.682353 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.678380    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-podinfo" pod "b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" (UID: "b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:38.698851 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.694016    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-kube-api-access-z274x" pod "b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" (UID: "b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff"). InnerVolumeSpecName "kube-api-access-z274x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:38.698851 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.694081    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-token" pod "b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" (UID: "b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:38.698851 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.696525    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-envoy" pod "b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" (UID: "b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:38.714890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.714856    5428 generic.go:356] "Generic (PLEG): container finished" podID="b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" containerID="e45987552fa8976f9c466a7e8f0717443c9563e07e22dae211aced093b1260f3" exitCode=0
Oct 29 23:03:38.715237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.714914    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"
Oct 29 23:03:38.715237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.714933    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll" event={"ID":"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff","Type":"ContainerDied","Data":"e45987552fa8976f9c466a7e8f0717443c9563e07e22dae211aced093b1260f3"}
Oct 29 23:03:38.715237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.714991    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll" event={"ID":"b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff","Type":"ContainerDied","Data":"3186fa3fddc617452fc877edbc475a63d61f08668289be9d7036479c2e126ad1"}
Oct 29 23:03:38.715237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.715016    5428 scope.go:122] "RemoveContainer" containerID="e45987552fa8976f9c466a7e8f0717443c9563e07e22dae211aced093b1260f3"
Oct 29 23:03:38.732044 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.728173    5428 scope.go:122] "RemoveContainer" containerID="f19ed61957bd77efa472c6ef591da14dac3b9e8abe6fb8c4e652edcc866c7d4e"
Oct 29 23:03:38.743269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.743242    5428 scope.go:122] "RemoveContainer" containerID="e45987552fa8976f9c466a7e8f0717443c9563e07e22dae211aced093b1260f3"
Oct 29 23:03:38.743622 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:03:38.743598    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e45987552fa8976f9c466a7e8f0717443c9563e07e22dae211aced093b1260f3\": container with ID starting with e45987552fa8976f9c466a7e8f0717443c9563e07e22dae211aced093b1260f3 not found: ID does not exist" containerID="e45987552fa8976f9c466a7e8f0717443c9563e07e22dae211aced093b1260f3"
Oct 29 23:03:38.743723 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.743628    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e45987552fa8976f9c466a7e8f0717443c9563e07e22dae211aced093b1260f3"} err="failed to get container status \"e45987552fa8976f9c466a7e8f0717443c9563e07e22dae211aced093b1260f3\": rpc error: code = NotFound desc = could not find container \"e45987552fa8976f9c466a7e8f0717443c9563e07e22dae211aced093b1260f3\": container with ID starting with e45987552fa8976f9c466a7e8f0717443c9563e07e22dae211aced093b1260f3 not found: ID does not exist"
Oct 29 23:03:38.743723 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.743665    5428 scope.go:122] "RemoveContainer" containerID="f19ed61957bd77efa472c6ef591da14dac3b9e8abe6fb8c4e652edcc866c7d4e"
Oct 29 23:03:38.743941 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:03:38.743924    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19ed61957bd77efa472c6ef591da14dac3b9e8abe6fb8c4e652edcc866c7d4e\": container with ID starting with f19ed61957bd77efa472c6ef591da14dac3b9e8abe6fb8c4e652edcc866c7d4e not found: ID does not exist" containerID="f19ed61957bd77efa472c6ef591da14dac3b9e8abe6fb8c4e652edcc866c7d4e"
Oct 29 23:03:38.743987 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.743945    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19ed61957bd77efa472c6ef591da14dac3b9e8abe6fb8c4e652edcc866c7d4e"} err="failed to get container status \"f19ed61957bd77efa472c6ef591da14dac3b9e8abe6fb8c4e652edcc866c7d4e\": rpc error: code = NotFound desc = could not find container \"f19ed61957bd77efa472c6ef591da14dac3b9e8abe6fb8c4e652edcc866c7d4e\": container with ID starting with f19ed61957bd77efa472c6ef591da14dac3b9e8abe6fb8c4e652edcc866c7d4e not found: ID does not exist"
Oct 29 23:03:38.792227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.791597    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z274x\" (UniqueName: \"kubernetes.io/projected/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-kube-api-access-z274x\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:38.792227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.791641    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:38.792227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.791656    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:38.792227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.791670    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:38.792227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.791684    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:38.792227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.791697    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:38.792227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.791710    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:38.792227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.791723    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:38.792227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.791735    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:38.822822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.814271    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"]
Oct 29 23:03:38.827508 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.827480    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-79e6c631-openshift-default-5c68db7765-gfqll"]
Oct 29 23:03:38.983828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.979896    5428 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 29 23:03:38.983828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:38.979959    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8" podUID="b886f3b9-2571-4566-9c33-4fea56038612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 29 23:03:39.696099 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:39.696061    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"]
Oct 29 23:03:39.696360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:39.696331    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n" podUID="4c00e005-e8e5-4e90-9b0c-95d5d5de4952" containerName="istio-proxy" containerID="cri-o://7cad7c838a179abf312b3e0a69cfd4753e40ee732ffbaf6e8cf150eaa8535c03" gracePeriod=30
Oct 29 23:03:39.713710 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:39.713674    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"]
Oct 29 23:03:39.728419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:39.727388    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn" podUID="101ae9b3-7447-4a91-b895-913c9a96f288" containerName="istio-proxy" containerID="cri-o://a65733053bfe76faf24b4111e3e28b5c95ffb551cb39923b8ded2134bf8358a8" gracePeriod=30
Oct 29 23:03:39.806831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:39.804655    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:39.944332 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:39.944293    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:39.957648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:39.957568    5428 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 29 23:03:39.957648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:39.957620    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8" podUID="b886f3b9-2571-4566-9c33-4fea56038612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 29 23:03:40.001969 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.001934    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf"]
Oct 29 23:03:40.002762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.002739    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" containerName="istio-proxy"
Oct 29 23:03:40.002762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.002763    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" containerName="istio-proxy"
Oct 29 23:03:40.002966 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.002782    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" containerName="istio-proxy"
Oct 29 23:03:40.002966 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.002790    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" containerName="istio-proxy"
Oct 29 23:03:40.003074 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.003031    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" containerName="istio-proxy"
Oct 29 23:03:40.003074 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.003052    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" containerName="istio-proxy"
Oct 29 23:03:40.009032 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.009011    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf"
Oct 29 23:03:40.024637 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.024607    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7af1a5\""
Oct 29 23:03:40.030612 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.030585    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7af1a5-dockercfg-9khfv\""
Oct 29 23:03:40.036540 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.036507    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf"]
Oct 29 23:03:40.090930 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.090897    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:40.094067 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.094038    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2"]
Oct 29 23:03:40.099093 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.099072    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2"
Oct 29 23:03:40.116946 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.116920    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2"]
Oct 29 23:03:40.118042 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.118017    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bc23a8-4f83-45d3-842d-08ec70fa41c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf\" (UID: \"21bc23a8-4f83-45d3-842d-08ec70fa41c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf"
Oct 29 23:03:40.118278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.118250    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8qd4\" (UniqueName: \"kubernetes.io/projected/21bc23a8-4f83-45d3-842d-08ec70fa41c8-kube-api-access-j8qd4\") pod \"payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf\" (UID: \"21bc23a8-4f83-45d3-842d-08ec70fa41c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf"
Oct 29 23:03:40.272879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.236140    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8qd4\" (UniqueName: \"kubernetes.io/projected/21bc23a8-4f83-45d3-842d-08ec70fa41c8-kube-api-access-j8qd4\") pod \"payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf\" (UID: \"21bc23a8-4f83-45d3-842d-08ec70fa41c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf"
Oct 29 23:03:40.273944 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.273906    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8lmz\" (UniqueName: \"kubernetes.io/projected/790e3088-94e6-441f-9a94-515b0dbd77c0-kube-api-access-t8lmz\") pod \"payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2\" (UID: \"790e3088-94e6-441f-9a94-515b0dbd77c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2"
Oct 29 23:03:40.274169 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.274150    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790e3088-94e6-441f-9a94-515b0dbd77c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2\" (UID: \"790e3088-94e6-441f-9a94-515b0dbd77c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2"
Oct 29 23:03:40.290659 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.278218    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bc23a8-4f83-45d3-842d-08ec70fa41c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf\" (UID: \"21bc23a8-4f83-45d3-842d-08ec70fa41c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf"
Oct 29 23:03:40.313833 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.290622    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bc23a8-4f83-45d3-842d-08ec70fa41c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf\" (UID: \"21bc23a8-4f83-45d3-842d-08ec70fa41c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf"
Oct 29 23:03:40.372822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.370341    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8qd4\" (UniqueName: \"kubernetes.io/projected/21bc23a8-4f83-45d3-842d-08ec70fa41c8-kube-api-access-j8qd4\") pod \"payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf\" (UID: \"21bc23a8-4f83-45d3-842d-08ec70fa41c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf"
Oct 29 23:03:40.413824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.408589    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8lmz\" (UniqueName: \"kubernetes.io/projected/790e3088-94e6-441f-9a94-515b0dbd77c0-kube-api-access-t8lmz\") pod \"payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2\" (UID: \"790e3088-94e6-441f-9a94-515b0dbd77c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2"
Oct 29 23:03:40.413824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.408651    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790e3088-94e6-441f-9a94-515b0dbd77c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2\" (UID: \"790e3088-94e6-441f-9a94-515b0dbd77c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2"
Oct 29 23:03:40.413824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.409833    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790e3088-94e6-441f-9a94-515b0dbd77c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2\" (UID: \"790e3088-94e6-441f-9a94-515b0dbd77c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2"
Oct 29 23:03:40.453826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.452821    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8lmz\" (UniqueName: \"kubernetes.io/projected/790e3088-94e6-441f-9a94-515b0dbd77c0-kube-api-access-t8lmz\") pod \"payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2\" (UID: \"790e3088-94e6-441f-9a94-515b0dbd77c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2"
Oct 29 23:03:40.625707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.625668    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf"
Oct 29 23:03:40.738450 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.738415    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2"
Oct 29 23:03:40.828437 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.827777    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff" path="/var/lib/kubelet/pods/b7de2d2a-c088-4e76-8a0d-f4e3cb0215ff/volumes"
Oct 29 23:03:40.852968 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.852932    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:40.999211 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.991088    5428 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 29 23:03:40.999211 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.991138    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8" podUID="b886f3b9-2571-4566-9c33-4fea56038612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 29 23:03:40.999211 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:40.992436    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:41.092004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.091954    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf"]
Oct 29 23:03:41.129568 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.126438    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:41.253274 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.253201    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2"]
Oct 29 23:03:41.307862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.307781    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:41.326153 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.325763    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 23:03:41.441154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.439388    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-data\") pod \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") "
Oct 29 23:03:41.441154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.439485    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-envoy\") pod \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") "
Oct 29 23:03:41.441154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.439583    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-credential-socket\") pod \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") "
Oct 29 23:03:41.441154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.439736    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-workload-socket\") pod \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") "
Oct 29 23:03:41.441154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.439773    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istiod-ca-cert\") pod \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") "
Oct 29 23:03:41.441154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.439835    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-token\" (UniqueName: \"kubernetes.io/projected/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-token\") pod \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") "
Oct 29 23:03:41.441154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.439896    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86892f1f-cbce-48a4-80ae-48e4c0f88abf-kube-api-access-c4p7s\" (UniqueName: \"kubernetes.io/projected/86892f1f-cbce-48a4-80ae-48e4c0f88abf-kube-api-access-c4p7s\") pod \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") "
Oct 29 23:03:41.441154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.439969    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-podinfo\") pod \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") "
Oct 29 23:03:41.441154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.440105    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-workload-certs\") pod \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\" (UID: \"86892f1f-cbce-48a4-80ae-48e4c0f88abf\") "
Oct 29 23:03:41.454877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.453377    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-data" pod "86892f1f-cbce-48a4-80ae-48e4c0f88abf" (UID: "86892f1f-cbce-48a4-80ae-48e4c0f88abf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:41.458664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.458472    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-envoy" pod "86892f1f-cbce-48a4-80ae-48e4c0f88abf" (UID: "86892f1f-cbce-48a4-80ae-48e4c0f88abf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:41.459050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.458873    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-credential-socket" pod "86892f1f-cbce-48a4-80ae-48e4c0f88abf" (UID: "86892f1f-cbce-48a4-80ae-48e4c0f88abf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:41.459372 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.459185    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-workload-socket" pod "86892f1f-cbce-48a4-80ae-48e4c0f88abf" (UID: "86892f1f-cbce-48a4-80ae-48e4c0f88abf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:41.459889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.459597    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istiod-ca-cert" pod "86892f1f-cbce-48a4-80ae-48e4c0f88abf" (UID: "86892f1f-cbce-48a4-80ae-48e4c0f88abf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:41.475099 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.465359    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-workload-certs" pod "86892f1f-cbce-48a4-80ae-48e4c0f88abf" (UID: "86892f1f-cbce-48a4-80ae-48e4c0f88abf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:41.561298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.551088    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-token" pod "86892f1f-cbce-48a4-80ae-48e4c0f88abf" (UID: "86892f1f-cbce-48a4-80ae-48e4c0f88abf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:41.561298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.552702    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:41.561298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.552733    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:41.561298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.552751    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:41.561298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.552768    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:41.561298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.552782    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:41.561298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.552796    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:41.561298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.552826    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:41.617880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.617834    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86892f1f-cbce-48a4-80ae-48e4c0f88abf-kube-api-access-c4p7s" pod "86892f1f-cbce-48a4-80ae-48e4c0f88abf" (UID: "86892f1f-cbce-48a4-80ae-48e4c0f88abf"). InnerVolumeSpecName "kube-api-access-c4p7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:41.668898 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.654271    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-podinfo" pod "86892f1f-cbce-48a4-80ae-48e4c0f88abf" (UID: "86892f1f-cbce-48a4-80ae-48e4c0f88abf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:41.693280 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.677010    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:41.693280 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.689015    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4p7s\" (UniqueName: \"kubernetes.io/projected/86892f1f-cbce-48a4-80ae-48e4c0f88abf-kube-api-access-c4p7s\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:41.693280 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.689043    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86892f1f-cbce-48a4-80ae-48e4c0f88abf-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:41.850043 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.850010    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 23:03:41.860356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.860319    5428 generic.go:356] "Generic (PLEG): container finished" podID="b20a2142-6441-4846-8855-b4595d2fe746" containerID="79a1b80d99a5ad05884bf5ec2a2cfab7b78c1b200ff75974faad6081ff247e7f" exitCode=0
Oct 29 23:03:41.860633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.860465    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"
Oct 29 23:03:41.861094 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.860487    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z" event={"ID":"b20a2142-6441-4846-8855-b4595d2fe746","Type":"ContainerDied","Data":"79a1b80d99a5ad05884bf5ec2a2cfab7b78c1b200ff75974faad6081ff247e7f"}
Oct 29 23:03:41.861094 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.861009    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z" event={"ID":"b20a2142-6441-4846-8855-b4595d2fe746","Type":"ContainerDied","Data":"83c47e091796ea8eef41183a1775703b3d4fe837ef195a5153c47cbfd785d6f8"}
Oct 29 23:03:41.861094 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.861056    5428 scope.go:122] "RemoveContainer" containerID="79a1b80d99a5ad05884bf5ec2a2cfab7b78c1b200ff75974faad6081ff247e7f"
Oct 29 23:03:41.864529 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.864503    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2" event={"ID":"790e3088-94e6-441f-9a94-515b0dbd77c0","Type":"ContainerStarted","Data":"40df54d33ce85da3cba869446ba0c24ae4f2a571ef853359a9f8b1051e7a2ee7"}
Oct 29 23:03:41.866413 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.866301    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2"
Oct 29 23:03:41.876660 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.876635    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf" event={"ID":"21bc23a8-4f83-45d3-842d-08ec70fa41c8","Type":"ContainerStarted","Data":"c159f24d2f53cf611464584671b1b08ed4db164f0ec6f883e5d1cc0381e0c9ea"}
Oct 29 23:03:41.876845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.876827    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf" event={"ID":"21bc23a8-4f83-45d3-842d-08ec70fa41c8","Type":"ContainerStarted","Data":"5ca80997fedacfd354ab7b13bfd8c7f4c7cd9237fe315453b24d1f883136159e"}
Oct 29 23:03:41.878214 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.878194    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf"
Oct 29 23:03:41.902962 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.883186    5428 generic.go:356] "Generic (PLEG): container finished" podID="86892f1f-cbce-48a4-80ae-48e4c0f88abf" containerID="f1fd7a3a2fab6e67ffda72157861716c9386953c97576bc67b32628f2b0a7ff5" exitCode=0
Oct 29 23:03:41.902962 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.883430    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx" event={"ID":"86892f1f-cbce-48a4-80ae-48e4c0f88abf","Type":"ContainerDied","Data":"f1fd7a3a2fab6e67ffda72157861716c9386953c97576bc67b32628f2b0a7ff5"}
Oct 29 23:03:41.902962 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.883488    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx" event={"ID":"86892f1f-cbce-48a4-80ae-48e4c0f88abf","Type":"ContainerDied","Data":"0fed2c92b39a651931ba62a2283feb3d1dfa99dd0016b9b3cc6b19edaba7b5e7"}
Oct 29 23:03:41.902962 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.883592    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"
Oct 29 23:03:41.916409 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.916371    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-credential-socket\") pod \"b20a2142-6441-4846-8855-b4595d2fe746\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") "
Oct 29 23:03:41.916570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.916476    5428 generic.go:356] "Generic (PLEG): container finished" podID="e415acf5-67c8-47c1-aa03-df58dca41b3a" containerID="6f4f37945e2bebf1405ffe03edddc744f8e15819e70efe15093d46c7b957cb08" exitCode=0
Oct 29 23:03:41.916570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.916510    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20a2142-6441-4846-8855-b4595d2fe746-istio-token\" (UniqueName: \"kubernetes.io/projected/b20a2142-6441-4846-8855-b4595d2fe746-istio-token\") pod \"b20a2142-6441-4846-8855-b4595d2fe746\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") "
Oct 29 23:03:41.916570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.916557    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-workload-certs\") pod \"b20a2142-6441-4846-8855-b4595d2fe746\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") "
Oct 29 23:03:41.916743 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.916628    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm" event={"ID":"e415acf5-67c8-47c1-aa03-df58dca41b3a","Type":"ContainerDied","Data":"6f4f37945e2bebf1405ffe03edddc744f8e15819e70efe15093d46c7b957cb08"}
Oct 29 23:03:41.916743 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.916653    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-istio-data\") pod \"b20a2142-6441-4846-8855-b4595d2fe746\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") "
Oct 29 23:03:41.916743 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.916724    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-istio-envoy\") pod \"b20a2142-6441-4846-8855-b4595d2fe746\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") "
Oct 29 23:03:41.916979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.916783    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b20a2142-6441-4846-8855-b4595d2fe746-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20a2142-6441-4846-8855-b4595d2fe746-istio-podinfo\") pod \"b20a2142-6441-4846-8855-b4595d2fe746\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") "
Oct 29 23:03:41.916979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.916909    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-workload-socket\") pod \"b20a2142-6441-4846-8855-b4595d2fe746\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") "
Oct 29 23:03:41.917091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.917018    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20a2142-6441-4846-8855-b4595d2fe746-kube-api-access-pk52j\" (UniqueName: \"kubernetes.io/projected/b20a2142-6441-4846-8855-b4595d2fe746-kube-api-access-pk52j\") pod \"b20a2142-6441-4846-8855-b4595d2fe746\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") "
Oct 29 23:03:41.917091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.917083    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b20a2142-6441-4846-8855-b4595d2fe746-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20a2142-6441-4846-8855-b4595d2fe746-istiod-ca-cert\") pod \"b20a2142-6441-4846-8855-b4595d2fe746\" (UID: \"b20a2142-6441-4846-8855-b4595d2fe746\") "
Oct 29 23:03:41.917710 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.917671    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b20a2142-6441-4846-8855-b4595d2fe746-istiod-ca-cert" pod "b20a2142-6441-4846-8855-b4595d2fe746" (UID: "b20a2142-6441-4846-8855-b4595d2fe746"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:41.919204 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.917991    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-credential-socket" pod "b20a2142-6441-4846-8855-b4595d2fe746" (UID: "b20a2142-6441-4846-8855-b4595d2fe746"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:41.919204 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.918448    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-workload-socket" pod "b20a2142-6441-4846-8855-b4595d2fe746" (UID: "b20a2142-6441-4846-8855-b4595d2fe746"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:41.919980 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.919958    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:41.920163 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.920139    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20a2142-6441-4846-8855-b4595d2fe746-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:41.920247 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.920166    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:41.920323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.920303    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-workload-certs" pod "b20a2142-6441-4846-8855-b4595d2fe746" (UID: "b20a2142-6441-4846-8855-b4595d2fe746"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:41.920771 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.920702    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-istio-data" pod "b20a2142-6441-4846-8855-b4595d2fe746" (UID: "b20a2142-6441-4846-8855-b4595d2fe746"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:41.922923 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.922893    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20a2142-6441-4846-8855-b4595d2fe746-istio-token" pod "b20a2142-6441-4846-8855-b4595d2fe746" (UID: "b20a2142-6441-4846-8855-b4595d2fe746"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:41.923447 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.923409    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-istio-envoy" pod "b20a2142-6441-4846-8855-b4595d2fe746" (UID: "b20a2142-6441-4846-8855-b4595d2fe746"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:41.923924 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.923898    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b20a2142-6441-4846-8855-b4595d2fe746-istio-podinfo" pod "b20a2142-6441-4846-8855-b4595d2fe746" (UID: "b20a2142-6441-4846-8855-b4595d2fe746"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:41.928314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.928294    5428 scope.go:122] "RemoveContainer" containerID="79a1b80d99a5ad05884bf5ec2a2cfab7b78c1b200ff75974faad6081ff247e7f"
Oct 29 23:03:41.928396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.928316    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20a2142-6441-4846-8855-b4595d2fe746-kube-api-access-pk52j" pod "b20a2142-6441-4846-8855-b4595d2fe746" (UID: "b20a2142-6441-4846-8855-b4595d2fe746"). InnerVolumeSpecName "kube-api-access-pk52j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:41.928782 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:03:41.928761    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a1b80d99a5ad05884bf5ec2a2cfab7b78c1b200ff75974faad6081ff247e7f\": container with ID starting with 79a1b80d99a5ad05884bf5ec2a2cfab7b78c1b200ff75974faad6081ff247e7f not found: ID does not exist" containerID="79a1b80d99a5ad05884bf5ec2a2cfab7b78c1b200ff75974faad6081ff247e7f"
Oct 29 23:03:41.928874 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.928789    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a1b80d99a5ad05884bf5ec2a2cfab7b78c1b200ff75974faad6081ff247e7f"} err="failed to get container status \"79a1b80d99a5ad05884bf5ec2a2cfab7b78c1b200ff75974faad6081ff247e7f\": rpc error: code = NotFound desc = could not find container \"79a1b80d99a5ad05884bf5ec2a2cfab7b78c1b200ff75974faad6081ff247e7f\": container with ID starting with 79a1b80d99a5ad05884bf5ec2a2cfab7b78c1b200ff75974faad6081ff247e7f not found: ID does not exist"
Oct 29 23:03:41.928874 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.928822    5428 scope.go:122] "RemoveContainer" containerID="f1fd7a3a2fab6e67ffda72157861716c9386953c97576bc67b32628f2b0a7ff5"
Oct 29 23:03:41.941370 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.941047    5428 scope.go:122] "RemoveContainer" containerID="f1fd7a3a2fab6e67ffda72157861716c9386953c97576bc67b32628f2b0a7ff5"
Oct 29 23:03:41.941370 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:03:41.941365    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1fd7a3a2fab6e67ffda72157861716c9386953c97576bc67b32628f2b0a7ff5\": container with ID starting with f1fd7a3a2fab6e67ffda72157861716c9386953c97576bc67b32628f2b0a7ff5 not found: ID does not exist" containerID="f1fd7a3a2fab6e67ffda72157861716c9386953c97576bc67b32628f2b0a7ff5"
Oct 29 23:03:41.941557 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.941386    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fd7a3a2fab6e67ffda72157861716c9386953c97576bc67b32628f2b0a7ff5"} err="failed to get container status \"f1fd7a3a2fab6e67ffda72157861716c9386953c97576bc67b32628f2b0a7ff5\": rpc error: code = NotFound desc = could not find container \"f1fd7a3a2fab6e67ffda72157861716c9386953c97576bc67b32628f2b0a7ff5\": container with ID starting with f1fd7a3a2fab6e67ffda72157861716c9386953c97576bc67b32628f2b0a7ff5 not found: ID does not exist"
Oct 29 23:03:41.949839 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.948653    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2" podStartSLOduration=1.9486352710000001 podStartE2EDuration="1.948635271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:41.944535662 +0000 UTC m=+1225.287484407" watchObservedRunningTime="2026-10-29 23:03:41.948635271 +0000 UTC m=+1225.291584017"
Oct 29 23:03:41.958368 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.958340    5428 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 29 23:03:41.958461 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:41.958398    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8" podUID="b886f3b9-2571-4566-9c33-4fea56038612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 29 23:03:42.001556 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.001491    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 23:03:42.020719 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.020667    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf" podStartSLOduration=3.020652638 podStartE2EDuration="3.020652638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:42.018909912 +0000 UTC m=+1225.361858657" watchObservedRunningTime="2026-10-29 23:03:42.020652638 +0000 UTC m=+1225.363601382"
Oct 29 23:03:42.023289 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.023257    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20a2142-6441-4846-8855-b4595d2fe746-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.023289 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.023290    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.023454 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.023304    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.023454 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.023316    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20a2142-6441-4846-8855-b4595d2fe746-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.023454 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.023328    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20a2142-6441-4846-8855-b4595d2fe746-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.023454 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.023341    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk52j\" (UniqueName: \"kubernetes.io/projected/b20a2142-6441-4846-8855-b4595d2fe746-kube-api-access-pk52j\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.123849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.123817    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e415acf5-67c8-47c1-aa03-df58dca41b3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e415acf5-67c8-47c1-aa03-df58dca41b3a-istiod-ca-cert\") pod \"e415acf5-67c8-47c1-aa03-df58dca41b3a\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") "
Oct 29 23:03:42.124028 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.123865    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-credential-socket\") pod \"e415acf5-67c8-47c1-aa03-df58dca41b3a\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") "
Oct 29 23:03:42.124028 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.124016    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-workload-certs\") pod \"e415acf5-67c8-47c1-aa03-df58dca41b3a\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") "
Oct 29 23:03:42.124173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.124091    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-workload-socket\") pod \"e415acf5-67c8-47c1-aa03-df58dca41b3a\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") "
Oct 29 23:03:42.124173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.124110    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e415acf5-67c8-47c1-aa03-df58dca41b3a-kube-api-access-zwdnz\" (UniqueName: \"kubernetes.io/projected/e415acf5-67c8-47c1-aa03-df58dca41b3a-kube-api-access-zwdnz\") pod \"e415acf5-67c8-47c1-aa03-df58dca41b3a\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") "
Oct 29 23:03:42.124173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.124135    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-envoy\") pod \"e415acf5-67c8-47c1-aa03-df58dca41b3a\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") "
Oct 29 23:03:42.124354 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.124220    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-data\") pod \"e415acf5-67c8-47c1-aa03-df58dca41b3a\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") "
Oct 29 23:03:42.124354 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.124247    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-token\" (UniqueName: \"kubernetes.io/projected/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-token\") pod \"e415acf5-67c8-47c1-aa03-df58dca41b3a\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") "
Oct 29 23:03:42.124354 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.124279    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-podinfo\") pod \"e415acf5-67c8-47c1-aa03-df58dca41b3a\" (UID: \"e415acf5-67c8-47c1-aa03-df58dca41b3a\") "
Oct 29 23:03:42.124696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.124668    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-credential-socket" pod "e415acf5-67c8-47c1-aa03-df58dca41b3a" (UID: "e415acf5-67c8-47c1-aa03-df58dca41b3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:42.124766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.124712    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e415acf5-67c8-47c1-aa03-df58dca41b3a-istiod-ca-cert" pod "e415acf5-67c8-47c1-aa03-df58dca41b3a" (UID: "e415acf5-67c8-47c1-aa03-df58dca41b3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:42.125009 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.124990    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e415acf5-67c8-47c1-aa03-df58dca41b3a-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.125073 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.125010    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.125268 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.125247    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-workload-certs" pod "e415acf5-67c8-47c1-aa03-df58dca41b3a" (UID: "e415acf5-67c8-47c1-aa03-df58dca41b3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:42.125495 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.125409    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-data" pod "e415acf5-67c8-47c1-aa03-df58dca41b3a" (UID: "e415acf5-67c8-47c1-aa03-df58dca41b3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:42.125622 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.125531    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-workload-socket" pod "e415acf5-67c8-47c1-aa03-df58dca41b3a" (UID: "e415acf5-67c8-47c1-aa03-df58dca41b3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:42.128028 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.127988    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-envoy" pod "e415acf5-67c8-47c1-aa03-df58dca41b3a" (UID: "e415acf5-67c8-47c1-aa03-df58dca41b3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:42.133750 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.133726    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"]
Oct 29 23:03:42.138025 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.138001    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-rmmvx"]
Oct 29 23:03:42.155916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.155890    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-podinfo" pod "e415acf5-67c8-47c1-aa03-df58dca41b3a" (UID: "e415acf5-67c8-47c1-aa03-df58dca41b3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:42.156011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.155918    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e415acf5-67c8-47c1-aa03-df58dca41b3a-kube-api-access-zwdnz" pod "e415acf5-67c8-47c1-aa03-df58dca41b3a" (UID: "e415acf5-67c8-47c1-aa03-df58dca41b3a"). InnerVolumeSpecName "kube-api-access-zwdnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:42.156011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.155948    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-token" pod "e415acf5-67c8-47c1-aa03-df58dca41b3a" (UID: "e415acf5-67c8-47c1-aa03-df58dca41b3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:42.227110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.226506    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.227110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.226535    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.227110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.226546    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.227110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.226556    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwdnz\" (UniqueName: \"kubernetes.io/projected/e415acf5-67c8-47c1-aa03-df58dca41b3a-kube-api-access-zwdnz\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.227110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.226567    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.227110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.226576    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.227110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.226583    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e415acf5-67c8-47c1-aa03-df58dca41b3a-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:42.245545 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.245485    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"]
Oct 29 23:03:42.247125 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:03:42.247087    5428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20a2142_6441_4846_8855_b4595d2fe746.slice/crio-83c47e091796ea8eef41183a1775703b3d4fe837ef195a5153c47cbfd785d6f8\": RecentStats: unable to find data in memory cache]"
Oct 29 23:03:42.250516 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.250488    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-gxv5z"]
Oct 29 23:03:42.795574 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.795539    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86892f1f-cbce-48a4-80ae-48e4c0f88abf" path="/var/lib/kubelet/pods/86892f1f-cbce-48a4-80ae-48e4c0f88abf/volumes"
Oct 29 23:03:42.796273 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.796246    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b20a2142-6441-4846-8855-b4595d2fe746" path="/var/lib/kubelet/pods/b20a2142-6441-4846-8855-b4595d2fe746/volumes"
Oct 29 23:03:42.929589 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.929553    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2" event={"ID":"790e3088-94e6-441f-9a94-515b0dbd77c0","Type":"ContainerStarted","Data":"a91df94e7ae90fc3435474a537e694f273f2593be2944832938ec94a1b576547"}
Oct 29 23:03:42.931457 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.931430    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:42.932285 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.932266    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"
Oct 29 23:03:42.932399 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.932296    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm" event={"ID":"e415acf5-67c8-47c1-aa03-df58dca41b3a","Type":"ContainerDied","Data":"283f87cd8742df6518d4c7df7b72907d6f411d3bf17bd064133bc41926959173"}
Oct 29 23:03:42.932399 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.932349    5428 scope.go:122] "RemoveContainer" containerID="6f4f37945e2bebf1405ffe03edddc744f8e15819e70efe15093d46c7b957cb08"
Oct 29 23:03:42.957411 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.957385    5428 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 29 23:03:42.957535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:42.957435    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8" podUID="b886f3b9-2571-4566-9c33-4fea56038612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 29 23:03:43.006813 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.006753    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"]
Oct 29 23:03:43.016513 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.016484    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-zg8vm"]
Oct 29 23:03:43.051398 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.051327    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:43.098144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.098070    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls"]
Oct 29 23:03:43.099483 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.099449    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b20a2142-6441-4846-8855-b4595d2fe746" containerName="istio-proxy"
Oct 29 23:03:43.099483 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.099476    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b20a2142-6441-4846-8855-b4595d2fe746" containerName="istio-proxy"
Oct 29 23:03:43.099739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.099495    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e415acf5-67c8-47c1-aa03-df58dca41b3a" containerName="istio-proxy"
Oct 29 23:03:43.099739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.099516    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e415acf5-67c8-47c1-aa03-df58dca41b3a" containerName="istio-proxy"
Oct 29 23:03:43.099739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.099528    5428 cpu_manager.go:414] "containerMap: removing container" podUID="86892f1f-cbce-48a4-80ae-48e4c0f88abf" containerName="istio-proxy"
Oct 29 23:03:43.099739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.099537    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86892f1f-cbce-48a4-80ae-48e4c0f88abf" containerName="istio-proxy"
Oct 29 23:03:43.099739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.099723    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b20a2142-6441-4846-8855-b4595d2fe746" containerName="istio-proxy"
Oct 29 23:03:43.100121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.099750    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e415acf5-67c8-47c1-aa03-df58dca41b3a" containerName="istio-proxy"
Oct 29 23:03:43.100121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.099761    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="86892f1f-cbce-48a4-80ae-48e4c0f88abf" containerName="istio-proxy"
Oct 29 23:03:43.106128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.106102    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls"]
Oct 29 23:03:43.106229 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.106196    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls"
Oct 29 23:03:43.121087 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.121049    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b0c72c\""
Oct 29 23:03:43.121199 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.121180    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b0c72c-dockercfg-44whn\""
Oct 29 23:03:43.153834 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.153812    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj"]
Oct 29 23:03:43.158081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.158064    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj"
Oct 29 23:03:43.188221 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.188189    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:43.210868 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.201532    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj"]
Oct 29 23:03:43.268831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.264344    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk2xv\" (UniqueName: \"kubernetes.io/projected/a4ec35bb-8ae9-4800-87a9-70907abc1a09-kube-api-access-zk2xv\") pod \"payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj\" (UID: \"a4ec35bb-8ae9-4800-87a9-70907abc1a09\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj"
Oct 29 23:03:43.268831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.264530    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mr7c\" (UniqueName: \"kubernetes.io/projected/67b3bd3b-224d-47b3-aee2-145b94b2635b-kube-api-access-9mr7c\") pod \"payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls\" (UID: \"67b3bd3b-224d-47b3-aee2-145b94b2635b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls"
Oct 29 23:03:43.268831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.264679    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4ec35bb-8ae9-4800-87a9-70907abc1a09-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj\" (UID: \"a4ec35bb-8ae9-4800-87a9-70907abc1a09\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj"
Oct 29 23:03:43.268831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.264760    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b3bd3b-224d-47b3-aee2-145b94b2635b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls\" (UID: \"67b3bd3b-224d-47b3-aee2-145b94b2635b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls"
Oct 29 23:03:43.461843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.374718    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mr7c\" (UniqueName: \"kubernetes.io/projected/67b3bd3b-224d-47b3-aee2-145b94b2635b-kube-api-access-9mr7c\") pod \"payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls\" (UID: \"67b3bd3b-224d-47b3-aee2-145b94b2635b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls"
Oct 29 23:03:43.461843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.374845    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4ec35bb-8ae9-4800-87a9-70907abc1a09-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj\" (UID: \"a4ec35bb-8ae9-4800-87a9-70907abc1a09\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj"
Oct 29 23:03:43.461843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.374905    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b3bd3b-224d-47b3-aee2-145b94b2635b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls\" (UID: \"67b3bd3b-224d-47b3-aee2-145b94b2635b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls"
Oct 29 23:03:43.461843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.375005    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk2xv\" (UniqueName: \"kubernetes.io/projected/a4ec35bb-8ae9-4800-87a9-70907abc1a09-kube-api-access-zk2xv\") pod \"payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj\" (UID: \"a4ec35bb-8ae9-4800-87a9-70907abc1a09\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj"
Oct 29 23:03:43.461843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.396559    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4ec35bb-8ae9-4800-87a9-70907abc1a09-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj\" (UID: \"a4ec35bb-8ae9-4800-87a9-70907abc1a09\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj"
Oct 29 23:03:43.461843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.435514    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:43.506852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.501760    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk2xv\" (UniqueName: \"kubernetes.io/projected/a4ec35bb-8ae9-4800-87a9-70907abc1a09-kube-api-access-zk2xv\") pod \"payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj\" (UID: \"a4ec35bb-8ae9-4800-87a9-70907abc1a09\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj"
Oct 29 23:03:43.519865 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.513118    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b3bd3b-224d-47b3-aee2-145b94b2635b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls\" (UID: \"67b3bd3b-224d-47b3-aee2-145b94b2635b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls"
Oct 29 23:03:43.643824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.640850    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj"
Oct 29 23:03:43.644276 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.644211    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mr7c\" (UniqueName: \"kubernetes.io/projected/67b3bd3b-224d-47b3-aee2-145b94b2635b-kube-api-access-9mr7c\") pod \"payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls\" (UID: \"67b3bd3b-224d-47b3-aee2-145b94b2635b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls"
Oct 29 23:03:43.725751 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.723169    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls"
Oct 29 23:03:43.997793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:43.997766    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj"]
Oct 29 23:03:43.998528 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:03:43.998497    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ec35bb_8ae9_4800_87a9_70907abc1a09.slice/crio-836834baa08cc186162eb3102c98ee546cff816ebd88975296f6f70707109fd5 WatchSource:0}: Error finding container 836834baa08cc186162eb3102c98ee546cff816ebd88975296f6f70707109fd5: Status 404 returned error can't find the container with id 836834baa08cc186162eb3102c98ee546cff816ebd88975296f6f70707109fd5
Oct 29 23:03:44.051053 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.051013    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:44.053496 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.053465    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:44.054722 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.054695    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:03:44.099405 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.098157    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls"]
Oct 29 23:03:44.102719 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:03:44.102693    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b3bd3b_224d_47b3_aee2_145b94b2635b.slice/crio-e9458289cfcac60b919b1bd74ef278d6a0348c5861f085cdea887b43cd280af7 WatchSource:0}: Error finding container e9458289cfcac60b919b1bd74ef278d6a0348c5861f085cdea887b43cd280af7: Status 404 returned error can't find the container with id e9458289cfcac60b919b1bd74ef278d6a0348c5861f085cdea887b43cd280af7
Oct 29 23:03:44.315042 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.315010    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs"]
Oct 29 23:03:44.326473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.322872    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs"
Oct 29 23:03:44.341874 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.340629    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-10b990\""
Oct 29 23:03:44.343105 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.343078    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-10b990-dockercfg-fkfp5\""
Oct 29 23:03:44.386103 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.386069    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs"]
Oct 29 23:03:44.461491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.461455    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9khzp\" (UniqueName: \"kubernetes.io/projected/1b38f1ba-e4a0-441e-bf0e-aab36c7f62db-kube-api-access-9khzp\") pod \"payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs\" (UID: \"1b38f1ba-e4a0-441e-bf0e-aab36c7f62db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs"
Oct 29 23:03:44.461816 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.461772    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b38f1ba-e4a0-441e-bf0e-aab36c7f62db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs\" (UID: \"1b38f1ba-e4a0-441e-bf0e-aab36c7f62db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs"
Oct 29 23:03:44.503143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.503108    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p"]
Oct 29 23:03:44.507997 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.507976    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p"
Oct 29 23:03:44.528586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.528528    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p"]
Oct 29 23:03:44.563784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.563740    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9khzp\" (UniqueName: \"kubernetes.io/projected/1b38f1ba-e4a0-441e-bf0e-aab36c7f62db-kube-api-access-9khzp\") pod \"payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs\" (UID: \"1b38f1ba-e4a0-441e-bf0e-aab36c7f62db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs"
Oct 29 23:03:44.564037 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.564020    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b38f1ba-e4a0-441e-bf0e-aab36c7f62db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs\" (UID: \"1b38f1ba-e4a0-441e-bf0e-aab36c7f62db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs"
Oct 29 23:03:44.564769 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.564750    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b38f1ba-e4a0-441e-bf0e-aab36c7f62db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs\" (UID: \"1b38f1ba-e4a0-441e-bf0e-aab36c7f62db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs"
Oct 29 23:03:44.587148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.587119    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9khzp\" (UniqueName: \"kubernetes.io/projected/1b38f1ba-e4a0-441e-bf0e-aab36c7f62db-kube-api-access-9khzp\") pod \"payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs\" (UID: \"1b38f1ba-e4a0-441e-bf0e-aab36c7f62db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs"
Oct 29 23:03:44.665190 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.665156    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8dnn\" (UniqueName: \"kubernetes.io/projected/c35a2e47-ef8f-4c89-a7c2-1a469211e3a3-kube-api-access-d8dnn\") pod \"payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p\" (UID: \"c35a2e47-ef8f-4c89-a7c2-1a469211e3a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p"
Oct 29 23:03:44.665370 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.665219    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35a2e47-ef8f-4c89-a7c2-1a469211e3a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p\" (UID: \"c35a2e47-ef8f-4c89-a7c2-1a469211e3a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p"
Oct 29 23:03:44.683601 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.683573    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs"
Oct 29 23:03:44.768501 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.767349    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8dnn\" (UniqueName: \"kubernetes.io/projected/c35a2e47-ef8f-4c89-a7c2-1a469211e3a3-kube-api-access-d8dnn\") pod \"payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p\" (UID: \"c35a2e47-ef8f-4c89-a7c2-1a469211e3a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p"
Oct 29 23:03:44.768501 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.767425    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35a2e47-ef8f-4c89-a7c2-1a469211e3a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p\" (UID: \"c35a2e47-ef8f-4c89-a7c2-1a469211e3a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p"
Oct 29 23:03:44.770676 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.768942    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35a2e47-ef8f-4c89-a7c2-1a469211e3a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p\" (UID: \"c35a2e47-ef8f-4c89-a7c2-1a469211e3a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p"
Oct 29 23:03:44.786723 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.786658    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e415acf5-67c8-47c1-aa03-df58dca41b3a" path="/var/lib/kubelet/pods/e415acf5-67c8-47c1-aa03-df58dca41b3a/volumes"
Oct 29 23:03:44.801865 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.801778    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8dnn\" (UniqueName: \"kubernetes.io/projected/c35a2e47-ef8f-4c89-a7c2-1a469211e3a3-kube-api-access-d8dnn\") pod \"payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p\" (UID: \"c35a2e47-ef8f-4c89-a7c2-1a469211e3a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p"
Oct 29 23:03:44.824381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.824345    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p"
Oct 29 23:03:44.975553 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.974401    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj" event={"ID":"a4ec35bb-8ae9-4800-87a9-70907abc1a09","Type":"ContainerStarted","Data":"03cbae06db73c04c388648d020f19234404097a0eee24a33253da6dc7cc238f1"}
Oct 29 23:03:44.975553 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.974475    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj" event={"ID":"a4ec35bb-8ae9-4800-87a9-70907abc1a09","Type":"ContainerStarted","Data":"836834baa08cc186162eb3102c98ee546cff816ebd88975296f6f70707109fd5"}
Oct 29 23:03:44.975553 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.975494    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj"
Oct 29 23:03:44.979054 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.978992    5428 generic.go:356] "Generic (PLEG): container finished" podID="4c00e005-e8e5-4e90-9b0c-95d5d5de4952" containerID="7cad7c838a179abf312b3e0a69cfd4753e40ee732ffbaf6e8cf150eaa8535c03" exitCode=0
Oct 29 23:03:44.979214 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.979158    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n" event={"ID":"4c00e005-e8e5-4e90-9b0c-95d5d5de4952","Type":"ContainerDied","Data":"7cad7c838a179abf312b3e0a69cfd4753e40ee732ffbaf6e8cf150eaa8535c03"}
Oct 29 23:03:44.982007 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.981977    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls" event={"ID":"67b3bd3b-224d-47b3-aee2-145b94b2635b","Type":"ContainerStarted","Data":"7b069b9b85c42ef5348b95593627c3ca44a0a2d21685f5061106ef4bb3e11c74"}
Oct 29 23:03:44.982192 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.982170    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls" event={"ID":"67b3bd3b-224d-47b3-aee2-145b94b2635b","Type":"ContainerStarted","Data":"e9458289cfcac60b919b1bd74ef278d6a0348c5861f085cdea887b43cd280af7"}
Oct 29 23:03:44.983078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.983056    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls"
Oct 29 23:03:44.985227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.985204    5428 generic.go:356] "Generic (PLEG): container finished" podID="101ae9b3-7447-4a91-b895-913c9a96f288" containerID="a65733053bfe76faf24b4111e3e28b5c95ffb551cb39923b8ded2134bf8358a8" exitCode=0
Oct 29 23:03:44.986128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:44.986092    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn" event={"ID":"101ae9b3-7447-4a91-b895-913c9a96f288","Type":"ContainerDied","Data":"a65733053bfe76faf24b4111e3e28b5c95ffb551cb39923b8ded2134bf8358a8"}
Oct 29 23:03:45.006794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.006763    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs"]
Oct 29 23:03:45.011660 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:03:45.011631    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b38f1ba_e4a0_441e_bf0e_aab36c7f62db.slice/crio-ec2357201805e74d09bee30573c04a3cab31013ac557e722876600b443480ac5 WatchSource:0}: Error finding container ec2357201805e74d09bee30573c04a3cab31013ac557e722876600b443480ac5: Status 404 returned error can't find the container with id ec2357201805e74d09bee30573c04a3cab31013ac557e722876600b443480ac5
Oct 29 23:03:45.022425 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.021762    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj" podStartSLOduration=2.021742749 podStartE2EDuration="2.021742749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:45.01612874 +0000 UTC m=+1228.359077486" watchObservedRunningTime="2026-10-29 23:03:45.021742749 +0000 UTC m=+1228.364691496"
Oct 29 23:03:45.052395 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.052367    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 23:03:45.055938 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.055913    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 23:03:45.074381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.074330    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls" podStartSLOduration=2.074315715 podStartE2EDuration="2.074315715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:45.06815823 +0000 UTC m=+1228.411106978" watchObservedRunningTime="2026-10-29 23:03:45.074315715 +0000 UTC m=+1228.417264460"
Oct 29 23:03:45.144795 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.144760    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p"]
Oct 29 23:03:45.144988 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:03:45.144872    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35a2e47_ef8f_4c89_a7c2_1a469211e3a3.slice/crio-c006171ffbc304d512b32f38af22f18349353c529e9c4e075a4eda28b357d099 WatchSource:0}: Error finding container c006171ffbc304d512b32f38af22f18349353c529e9c4e075a4eda28b357d099: Status 404 returned error can't find the container with id c006171ffbc304d512b32f38af22f18349353c529e9c4e075a4eda28b357d099
Oct 29 23:03:45.177255 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.177220    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-kube-api-access-4dqf6\" (UniqueName: \"kubernetes.io/projected/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-kube-api-access-4dqf6\") pod \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") "
Oct 29 23:03:45.177417 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.177261    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101ae9b3-7447-4a91-b895-913c9a96f288-kube-api-access-nmrdp\" (UniqueName: \"kubernetes.io/projected/101ae9b3-7447-4a91-b895-913c9a96f288-kube-api-access-nmrdp\") pod \"101ae9b3-7447-4a91-b895-913c9a96f288\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") "
Oct 29 23:03:45.177417 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.177299    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-workload-certs\") pod \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") "
Oct 29 23:03:45.177417 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.177335    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101ae9b3-7447-4a91-b895-913c9a96f288-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101ae9b3-7447-4a91-b895-913c9a96f288-istiod-ca-cert\") pod \"101ae9b3-7447-4a91-b895-913c9a96f288\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") "
Oct 29 23:03:45.177597 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.177454    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-envoy\") pod \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") "
Oct 29 23:03:45.177597 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.177547    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/101ae9b3-7447-4a91-b895-913c9a96f288-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101ae9b3-7447-4a91-b895-913c9a96f288-istio-podinfo\") pod \"101ae9b3-7447-4a91-b895-913c9a96f288\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") "
Oct 29 23:03:45.177708 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.177642    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-workload-certs" pod "4c00e005-e8e5-4e90-9b0c-95d5d5de4952" (UID: "4c00e005-e8e5-4e90-9b0c-95d5d5de4952"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:45.177708 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.177673    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-workload-certs\") pod \"101ae9b3-7447-4a91-b895-913c9a96f288\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") "
Oct 29 23:03:45.178159 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.177790    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-podinfo\") pod \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") "
Oct 29 23:03:45.178597 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.177903    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-workload-certs" pod "101ae9b3-7447-4a91-b895-913c9a96f288" (UID: "101ae9b3-7447-4a91-b895-913c9a96f288"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:45.178597 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.178476    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101ae9b3-7447-4a91-b895-913c9a96f288-istiod-ca-cert" pod "101ae9b3-7447-4a91-b895-913c9a96f288" (UID: "101ae9b3-7447-4a91-b895-913c9a96f288"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:45.178779 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.178601    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-istio-envoy\") pod \"101ae9b3-7447-4a91-b895-913c9a96f288\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") "
Oct 29 23:03:45.178779 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.178642    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-istio-data\") pod \"101ae9b3-7447-4a91-b895-913c9a96f288\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") "
Oct 29 23:03:45.178779 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.178681    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-token\" (UniqueName: \"kubernetes.io/projected/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-token\") pod \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") "
Oct 29 23:03:45.178779 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.178728    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istiod-ca-cert\") pod \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") "
Oct 29 23:03:45.178779 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.178759    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101ae9b3-7447-4a91-b895-913c9a96f288-istio-token\" (UniqueName: \"kubernetes.io/projected/101ae9b3-7447-4a91-b895-913c9a96f288-istio-token\") pod \"101ae9b3-7447-4a91-b895-913c9a96f288\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") "
Oct 29 23:03:45.182953 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.179046    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-istio-data" pod "101ae9b3-7447-4a91-b895-913c9a96f288" (UID: "101ae9b3-7447-4a91-b895-913c9a96f288"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:45.182953 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.180088    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101ae9b3-7447-4a91-b895-913c9a96f288-kube-api-access-nmrdp" pod "101ae9b3-7447-4a91-b895-913c9a96f288" (UID: "101ae9b3-7447-4a91-b895-913c9a96f288"). InnerVolumeSpecName "kube-api-access-nmrdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:45.183101 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.180235    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-kube-api-access-4dqf6" pod "4c00e005-e8e5-4e90-9b0c-95d5d5de4952" (UID: "4c00e005-e8e5-4e90-9b0c-95d5d5de4952"). InnerVolumeSpecName "kube-api-access-4dqf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:45.183101 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.180416    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istiod-ca-cert" pod "4c00e005-e8e5-4e90-9b0c-95d5d5de4952" (UID: "4c00e005-e8e5-4e90-9b0c-95d5d5de4952"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:45.183101 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.181203    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-envoy" pod "4c00e005-e8e5-4e90-9b0c-95d5d5de4952" (UID: "4c00e005-e8e5-4e90-9b0c-95d5d5de4952"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:45.183101 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.183094    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-workload-socket\") pod \"101ae9b3-7447-4a91-b895-913c9a96f288\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") "
Oct 29 23:03:45.184050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.183308    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-workload-socket" pod "101ae9b3-7447-4a91-b895-913c9a96f288" (UID: "101ae9b3-7447-4a91-b895-913c9a96f288"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:45.184050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.183351    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-workload-socket\") pod \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") "
Oct 29 23:03:45.184050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.183549    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-workload-socket" pod "4c00e005-e8e5-4e90-9b0c-95d5d5de4952" (UID: "4c00e005-e8e5-4e90-9b0c-95d5d5de4952"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:45.184050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.183659    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-credential-socket\") pod \"101ae9b3-7447-4a91-b895-913c9a96f288\" (UID: \"101ae9b3-7447-4a91-b895-913c9a96f288\") "
Oct 29 23:03:45.184050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.183875    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-credential-socket" pod "101ae9b3-7447-4a91-b895-913c9a96f288" (UID: "101ae9b3-7447-4a91-b895-913c9a96f288"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:45.184050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.183967    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-data\") pod \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") "
Oct 29 23:03:45.184050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.184046    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-credential-socket\") pod \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\" (UID: \"4c00e005-e8e5-4e90-9b0c-95d5d5de4952\") "
Oct 29 23:03:45.184414 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.184240    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-credential-socket" pod "4c00e005-e8e5-4e90-9b0c-95d5d5de4952" (UID: "4c00e005-e8e5-4e90-9b0c-95d5d5de4952"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:45.185029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.185005    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.185121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.185031    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.185121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.185045    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.185121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.185059    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.185121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.185072    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.185121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.185087    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.185121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.185100    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.185121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.185114    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dqf6\" (UniqueName: \"kubernetes.io/projected/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-kube-api-access-4dqf6\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.185491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.185130    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmrdp\" (UniqueName: \"kubernetes.io/projected/101ae9b3-7447-4a91-b895-913c9a96f288-kube-api-access-nmrdp\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.185491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.185143    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.185491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.185156    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101ae9b3-7447-4a91-b895-913c9a96f288-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.185491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.185169    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.186338 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.186302    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101ae9b3-7447-4a91-b895-913c9a96f288-istio-token" pod "101ae9b3-7447-4a91-b895-913c9a96f288" (UID: "101ae9b3-7447-4a91-b895-913c9a96f288"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:45.186425 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.186344    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-istio-envoy" pod "101ae9b3-7447-4a91-b895-913c9a96f288" (UID: "101ae9b3-7447-4a91-b895-913c9a96f288"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:45.186595 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.186567    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-token" pod "4c00e005-e8e5-4e90-9b0c-95d5d5de4952" (UID: "4c00e005-e8e5-4e90-9b0c-95d5d5de4952"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:45.186883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.186858    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/101ae9b3-7447-4a91-b895-913c9a96f288-istio-podinfo" pod "101ae9b3-7447-4a91-b895-913c9a96f288" (UID: "101ae9b3-7447-4a91-b895-913c9a96f288"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:45.186988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.186968    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-podinfo" pod "4c00e005-e8e5-4e90-9b0c-95d5d5de4952" (UID: "4c00e005-e8e5-4e90-9b0c-95d5d5de4952"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:45.191943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.191913    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-data" pod "4c00e005-e8e5-4e90-9b0c-95d5d5de4952" (UID: "4c00e005-e8e5-4e90-9b0c-95d5d5de4952"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:45.287341 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.287303    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.287341 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.287334    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101ae9b3-7447-4a91-b895-913c9a96f288-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.287341 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.287344    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.287341 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.287352    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/101ae9b3-7447-4a91-b895-913c9a96f288-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.287609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.287360    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c00e005-e8e5-4e90-9b0c-95d5d5de4952-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.287609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.287367    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101ae9b3-7447-4a91-b895-913c9a96f288-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:03:45.322624 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.322546    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:45.998337 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.997076    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"
Oct 29 23:03:45.998337 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.997096    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn" event={"ID":"101ae9b3-7447-4a91-b895-913c9a96f288","Type":"ContainerDied","Data":"05b3478fd223a38743df7d4ae1e5d179988d2b59b0c6aa088d4e658956a76a6e"}
Oct 29 23:03:45.998337 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:45.997169    5428 scope.go:122] "RemoveContainer" containerID="a65733053bfe76faf24b4111e3e28b5c95ffb551cb39923b8ded2134bf8358a8"
Oct 29 23:03:46.004310 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:46.004287    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"
Oct 29 23:03:46.004447 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:46.004280    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n" event={"ID":"4c00e005-e8e5-4e90-9b0c-95d5d5de4952","Type":"ContainerDied","Data":"d46e74fbba462cb930ef6e2fc21b4558db0fb581a8aa34d62be0aa87e13103b9"}
Oct 29 23:03:46.006464 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:46.006435    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p" event={"ID":"c35a2e47-ef8f-4c89-a7c2-1a469211e3a3","Type":"ContainerStarted","Data":"c006171ffbc304d512b32f38af22f18349353c529e9c4e075a4eda28b357d099"}
Oct 29 23:03:46.009144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:46.008896    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs" event={"ID":"1b38f1ba-e4a0-441e-bf0e-aab36c7f62db","Type":"ContainerStarted","Data":"ec2357201805e74d09bee30573c04a3cab31013ac557e722876600b443480ac5"}
Oct 29 23:03:46.054639 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:46.054004    5428 scope.go:122] "RemoveContainer" containerID="7cad7c838a179abf312b3e0a69cfd4753e40ee732ffbaf6e8cf150eaa8535c03"
Oct 29 23:03:46.070827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:46.070785    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"]
Oct 29 23:03:46.082278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:46.082249    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-tcgkn"]
Oct 29 23:03:46.117307 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:46.117277    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"]
Oct 29 23:03:46.122287 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:46.122254    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-khs7n"]
Oct 29 23:03:46.782526 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:46.782439    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101ae9b3-7447-4a91-b895-913c9a96f288" path="/var/lib/kubelet/pods/101ae9b3-7447-4a91-b895-913c9a96f288/volumes"
Oct 29 23:03:46.783035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:46.783020    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c00e005-e8e5-4e90-9b0c-95d5d5de4952" path="/var/lib/kubelet/pods/4c00e005-e8e5-4e90-9b0c-95d5d5de4952/volumes"
Oct 29 23:03:47.019854 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:47.019816    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs" event={"ID":"1b38f1ba-e4a0-441e-bf0e-aab36c7f62db","Type":"ContainerStarted","Data":"4df674bbdae857d61bc62b3ae6b2cf1a5065fcc5718055e1cfd21f38337e20e2"}
Oct 29 23:03:47.023790 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:47.023758    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p" event={"ID":"c35a2e47-ef8f-4c89-a7c2-1a469211e3a3","Type":"ContainerStarted","Data":"99ecf17535366f704b73ce8efff50b3cc4aefc78f20e0e065e0cdd321b20b566"}
Oct 29 23:03:47.023938 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:47.023922    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p"
Oct 29 23:03:47.068294 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:47.068210    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs" podStartSLOduration=1.672117128 podStartE2EDuration="3.068197343s" totalImagesPullingTime="1.396080215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:47.066238617 +0000 UTC m=+1230.409187394" watchObservedRunningTime="2026-10-29 23:03:47.068197343 +0000 UTC m=+1230.411146084"
Oct 29 23:03:47.118049 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:47.118003    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p" podStartSLOduration=1.817956987 podStartE2EDuration="3.117989946s" totalImagesPullingTime="1.300032959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:47.114165711 +0000 UTC m=+1230.457114457" watchObservedRunningTime="2026-10-29 23:03:47.117989946 +0000 UTC m=+1230.460938691"
Oct 29 23:03:47.339652 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:47.339551    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:47.394822 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:03:47.390354    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:03:48.031006 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:48.030974    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs"
Oct 29 23:03:48.194862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:48.194833    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:48.338790 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:48.338714    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:48.558101 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:48.558067    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:49.606446 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:49.606414    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:50.956639 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:50.956602    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:51.179918 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:51.179880    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:51.701141 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:51.701097    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d"]
Oct 29 23:03:51.701893 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:51.701877    5428 cpu_manager.go:414] "containerMap: removing container" podUID="4c00e005-e8e5-4e90-9b0c-95d5d5de4952" containerName="istio-proxy"
Oct 29 23:03:51.701963 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:51.701894    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c00e005-e8e5-4e90-9b0c-95d5d5de4952" containerName="istio-proxy"
Oct 29 23:03:51.701963 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:51.701929    5428 cpu_manager.go:414] "containerMap: removing container" podUID="101ae9b3-7447-4a91-b895-913c9a96f288" containerName="istio-proxy"
Oct 29 23:03:51.701963 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:51.701934    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101ae9b3-7447-4a91-b895-913c9a96f288" containerName="istio-proxy"
Oct 29 23:03:51.702067 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:51.702039    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c00e005-e8e5-4e90-9b0c-95d5d5de4952" containerName="istio-proxy"
Oct 29 23:03:51.702067 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:51.702054    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="101ae9b3-7447-4a91-b895-913c9a96f288" containerName="istio-proxy"
Oct 29 23:03:51.707420 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:51.707395    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d"
Oct 29 23:03:51.729728 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:51.729701    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d"]
Oct 29 23:03:51.734598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:51.734566    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-44a4e4-dockercfg-grb62\""
Oct 29 23:03:51.734700 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:51.734612    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-44a4e4\""
Oct 29 23:03:51.835698 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:51.835666    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfvhg\" (UniqueName: \"kubernetes.io/projected/be26e3f8-a847-49bc-aa52-c37190c3c2c6-kube-api-access-tfvhg\") pod \"payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d\" (UID: \"be26e3f8-a847-49bc-aa52-c37190c3c2c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d"
Oct 29 23:03:51.835966 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:51.835830    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be26e3f8-a847-49bc-aa52-c37190c3c2c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d\" (UID: \"be26e3f8-a847-49bc-aa52-c37190c3c2c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d"
Oct 29 23:03:51.938408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:51.938359    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfvhg\" (UniqueName: \"kubernetes.io/projected/be26e3f8-a847-49bc-aa52-c37190c3c2c6-kube-api-access-tfvhg\") pod \"payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d\" (UID: \"be26e3f8-a847-49bc-aa52-c37190c3c2c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d"
Oct 29 23:03:51.938582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:51.938447    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be26e3f8-a847-49bc-aa52-c37190c3c2c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d\" (UID: \"be26e3f8-a847-49bc-aa52-c37190c3c2c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d"
Oct 29 23:03:51.939317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:51.939291    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be26e3f8-a847-49bc-aa52-c37190c3c2c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d\" (UID: \"be26e3f8-a847-49bc-aa52-c37190c3c2c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d"
Oct 29 23:03:51.962438 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:51.962363    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfvhg\" (UniqueName: \"kubernetes.io/projected/be26e3f8-a847-49bc-aa52-c37190c3c2c6-kube-api-access-tfvhg\") pod \"payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d\" (UID: \"be26e3f8-a847-49bc-aa52-c37190c3c2c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d"
Oct 29 23:03:52.023355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:52.023328    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d"
Oct 29 23:03:52.185786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:52.181751    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f"]
Oct 29 23:03:52.189386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:52.189355    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f"
Oct 29 23:03:52.205533 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:52.204549    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f"]
Oct 29 23:03:52.350861 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:52.350773    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f\" (UID: \"c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f"
Oct 29 23:03:52.350861 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:52.350847    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6r5w\" (UniqueName: \"kubernetes.io/projected/c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2-kube-api-access-p6r5w\") pod \"payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f\" (UID: \"c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f"
Oct 29 23:03:52.358732 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:52.358704    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d"]
Oct 29 23:03:52.361925 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:03:52.361895    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe26e3f8_a847_49bc_aa52_c37190c3c2c6.slice/crio-228c5d07ffd74392985334750967989eb6c34a0cca094cd3997a87b8d82dcf14 WatchSource:0}: Error finding container 228c5d07ffd74392985334750967989eb6c34a0cca094cd3997a87b8d82dcf14: Status 404 returned error can't find the container with id 228c5d07ffd74392985334750967989eb6c34a0cca094cd3997a87b8d82dcf14
Oct 29 23:03:52.452932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:52.452900    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f\" (UID: \"c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f"
Oct 29 23:03:52.453029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:52.452948    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6r5w\" (UniqueName: \"kubernetes.io/projected/c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2-kube-api-access-p6r5w\") pod \"payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f\" (UID: \"c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f"
Oct 29 23:03:52.453548 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:52.453528    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f\" (UID: \"c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f"
Oct 29 23:03:52.476929 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:52.476893    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6r5w\" (UniqueName: \"kubernetes.io/projected/c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2-kube-api-access-p6r5w\") pod \"payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f\" (UID: \"c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f"
Oct 29 23:03:52.521218 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:52.521187    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f"
Oct 29 23:03:52.836879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:52.836848    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f"]
Oct 29 23:03:52.837423 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:03:52.837402    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c5fcb5_e3e6_4bcb_9453_6a7c9868b2e2.slice/crio-1167827c92f76736c0789c04032030227016371acaa1fd71ff9b2b6dbf19e6d9 WatchSource:0}: Error finding container 1167827c92f76736c0789c04032030227016371acaa1fd71ff9b2b6dbf19e6d9: Status 404 returned error can't find the container with id 1167827c92f76736c0789c04032030227016371acaa1fd71ff9b2b6dbf19e6d9
Oct 29 23:03:52.995834 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:52.994306    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:53.089076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:53.088973    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d" event={"ID":"be26e3f8-a847-49bc-aa52-c37190c3c2c6","Type":"ContainerStarted","Data":"e2bc5c9cbc65f7157de8abb6c95f8a18d7d9f52a4eb847b379aca3520d6fee7b"}
Oct 29 23:03:53.089076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:53.089050    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d" event={"ID":"be26e3f8-a847-49bc-aa52-c37190c3c2c6","Type":"ContainerStarted","Data":"228c5d07ffd74392985334750967989eb6c34a0cca094cd3997a87b8d82dcf14"}
Oct 29 23:03:53.089292 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:53.089080    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d"
Oct 29 23:03:53.091459 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:53.091333    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f" event={"ID":"c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2","Type":"ContainerStarted","Data":"c1ceb58dfdf80a91eee29c00d16810df3e0db3c285d76c1c9a903c24d12f4e20"}
Oct 29 23:03:53.091580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:53.091421    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f" event={"ID":"c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2","Type":"ContainerStarted","Data":"1167827c92f76736c0789c04032030227016371acaa1fd71ff9b2b6dbf19e6d9"}
Oct 29 23:03:53.091580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:53.091497    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f"
Oct 29 23:03:53.143843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:53.139057    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d" podStartSLOduration=2.139037776 podStartE2EDuration="2.139037776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:53.132006134 +0000 UTC m=+1236.474954880" watchObservedRunningTime="2026-10-29 23:03:53.139037776 +0000 UTC m=+1236.481986522"
Oct 29 23:03:53.186831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:53.185618    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f" podStartSLOduration=1.185600633 podStartE2EDuration="1.185600633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:53.176315861 +0000 UTC m=+1236.519264608" watchObservedRunningTime="2026-10-29 23:03:53.185600633 +0000 UTC m=+1236.528549374"
Oct 29 23:03:53.963184 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:53.963154    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2"
Oct 29 23:03:53.963349 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:53.963239    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf"
Oct 29 23:03:54.807421 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:54.807387    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:56.699868 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:56.699837    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:57.027707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:57.027625    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj"
Oct 29 23:03:57.027707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:57.027705    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls"
Oct 29 23:03:58.032625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:58.032594    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p"
Oct 29 23:03:58.998246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:58.998213    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:59.053523 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:59.051959    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs"
Oct 29 23:03:59.927678 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:03:59.927637    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:00.042195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:00.041585    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:01.795634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:01.795598    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:03.693573 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:03.693536    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:04.105114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:04.105033    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d"
Oct 29 23:04:04.105114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:04.105112    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f"
Oct 29 23:04:05.545259 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:05.545229    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:07.044080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:07.044049    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:07.223029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:07.222981    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:07.340202 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:07.340108    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:07.705607 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:07.705555    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:07.964731 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:07.964645    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:08.270917 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:08.270783    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:08.547114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:08.547035    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:09.751860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:09.751828    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:10.262780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:10.262741    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:10.510048 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:10.510014    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:10.865722 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:10.865688    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:12.451814 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.451765    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"]
Oct 29 23:04:12.457105 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.457087    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"
Oct 29 23:04:12.494113 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.494080    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7af1a5-serving-cert\""
Oct 29 23:04:12.500311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.500199    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"]
Oct 29 23:04:12.554257 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.554220    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:12.603448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.603407    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6112e33d-bd66-4ebc-8fb0-08a595027411-metrics-certs\") pod \"maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n\" (UID: \"6112e33d-bd66-4ebc-8fb0-08a595027411\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"
Oct 29 23:04:12.603690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.603590    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6112e33d-bd66-4ebc-8fb0-08a595027411-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n\" (UID: \"6112e33d-bd66-4ebc-8fb0-08a595027411\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"
Oct 29 23:04:12.603690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.603623    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxcrj\" (UniqueName: \"kubernetes.io/projected/6112e33d-bd66-4ebc-8fb0-08a595027411-kube-api-access-zxcrj\") pod \"maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n\" (UID: \"6112e33d-bd66-4ebc-8fb0-08a595027411\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"
Oct 29 23:04:12.603795 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.603684    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6112e33d-bd66-4ebc-8fb0-08a595027411-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n\" (UID: \"6112e33d-bd66-4ebc-8fb0-08a595027411\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"
Oct 29 23:04:12.675551 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.675500    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:12.705364 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.705272    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6112e33d-bd66-4ebc-8fb0-08a595027411-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n\" (UID: \"6112e33d-bd66-4ebc-8fb0-08a595027411\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"
Oct 29 23:04:12.705364 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.705323    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxcrj\" (UniqueName: \"kubernetes.io/projected/6112e33d-bd66-4ebc-8fb0-08a595027411-kube-api-access-zxcrj\") pod \"maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n\" (UID: \"6112e33d-bd66-4ebc-8fb0-08a595027411\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"
Oct 29 23:04:12.705647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.705388    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6112e33d-bd66-4ebc-8fb0-08a595027411-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n\" (UID: \"6112e33d-bd66-4ebc-8fb0-08a595027411\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"
Oct 29 23:04:12.705647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.705612    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6112e33d-bd66-4ebc-8fb0-08a595027411-metrics-certs\") pod \"maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n\" (UID: \"6112e33d-bd66-4ebc-8fb0-08a595027411\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"
Oct 29 23:04:12.706120 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.706095    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6112e33d-bd66-4ebc-8fb0-08a595027411-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n\" (UID: \"6112e33d-bd66-4ebc-8fb0-08a595027411\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"
Oct 29 23:04:12.708887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.708860    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6112e33d-bd66-4ebc-8fb0-08a595027411-metrics-certs\") pod \"maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n\" (UID: \"6112e33d-bd66-4ebc-8fb0-08a595027411\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"
Oct 29 23:04:12.709602 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.708864    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6112e33d-bd66-4ebc-8fb0-08a595027411-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n\" (UID: \"6112e33d-bd66-4ebc-8fb0-08a595027411\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"
Oct 29 23:04:12.728674 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.728635    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxcrj\" (UniqueName: \"kubernetes.io/projected/6112e33d-bd66-4ebc-8fb0-08a595027411-kube-api-access-zxcrj\") pod \"maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n\" (UID: \"6112e33d-bd66-4ebc-8fb0-08a595027411\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"
Oct 29 23:04:12.772146 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.772118    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"
Oct 29 23:04:12.806966 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:12.806937    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:13.092891 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:13.092856    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"]
Oct 29 23:04:13.093528 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:04:13.093503    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6112e33d_bd66_4ebc_8fb0_08a595027411.slice/crio-a08cbd7a86592cc6185f346aeeb1eb628cf1fb8209e4ff6be78ecb18189469e4 WatchSource:0}: Error finding container a08cbd7a86592cc6185f346aeeb1eb628cf1fb8209e4ff6be78ecb18189469e4: Status 404 returned error can't find the container with id a08cbd7a86592cc6185f346aeeb1eb628cf1fb8209e4ff6be78ecb18189469e4
Oct 29 23:04:13.282416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:13.282385    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n" event={"ID":"6112e33d-bd66-4ebc-8fb0-08a595027411","Type":"ContainerStarted","Data":"a08cbd7a86592cc6185f346aeeb1eb628cf1fb8209e4ff6be78ecb18189469e4"}
Oct 29 23:04:14.293653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:14.293611    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n" event={"ID":"6112e33d-bd66-4ebc-8fb0-08a595027411","Type":"ContainerStarted","Data":"ab6ad1c6ae3faeb9fe06cfac05f3427b2fc6be70f516aa6d2cb93921f260043f"}
Oct 29 23:04:14.294072 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:14.293679    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"
Oct 29 23:04:14.302831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:14.302762    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:14.345613 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:14.345556    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n" podStartSLOduration=1.970549109 podStartE2EDuration="2.34554145s" totalImagesPullingTime="374.992341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:14.34112697 +0000 UTC m=+1257.684075717" watchObservedRunningTime="2026-10-29 23:04:14.34554145 +0000 UTC m=+1257.688490194"
Oct 29 23:04:14.443358 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:14.443322    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:15.008771 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:15.008735    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:15.896959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:15.896924    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:16.015960 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.015920    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:16.074626 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.074590    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz"]
Oct 29 23:04:16.079622 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.079599    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz"
Oct 29 23:04:16.099990 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.099958    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz"]
Oct 29 23:04:16.105588 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.105564    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-41b12c\""
Oct 29 23:04:16.105734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.105565    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-41b12c-dockercfg-kv4t6\""
Oct 29 23:04:16.149175 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.149096    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:16.150448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.150419    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w"]
Oct 29 23:04:16.156508 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.156481    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w"
Oct 29 23:04:16.172272 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.172248    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w"]
Oct 29 23:04:16.187038 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.187007    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca44441-e031-4a00-8814-6b5e193b0f4a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz\" (UID: \"aca44441-e031-4a00-8814-6b5e193b0f4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz"
Oct 29 23:04:16.187324 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.187301    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8e1b51-65ba-47f7-b7f5-df5fb69a0203-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w\" (UID: \"7b8e1b51-65ba-47f7-b7f5-df5fb69a0203\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w"
Oct 29 23:04:16.187674 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.187531    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkmc5\" (UniqueName: \"kubernetes.io/projected/7b8e1b51-65ba-47f7-b7f5-df5fb69a0203-kube-api-access-vkmc5\") pod \"payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w\" (UID: \"7b8e1b51-65ba-47f7-b7f5-df5fb69a0203\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w"
Oct 29 23:04:16.187813 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.187770    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j48cc\" (UniqueName: \"kubernetes.io/projected/aca44441-e031-4a00-8814-6b5e193b0f4a-kube-api-access-j48cc\") pod \"payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz\" (UID: \"aca44441-e031-4a00-8814-6b5e193b0f4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz"
Oct 29 23:04:16.233265 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.233232    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"]
Oct 29 23:04:16.233437 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.233310    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"]
Oct 29 23:04:16.233538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.233506    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq" podUID="a8f4481b-afbf-41b9-b739-df14b6a3e77f" containerName="istio-proxy" containerID="cri-o://25bc92ffd492b08ef1b794e443d4c8214242944a2692f6a63fe05a7d5791565e" gracePeriod=30
Oct 29 23:04:16.233855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.233824    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752" podUID="9eeb8c61-ed7d-420a-b256-e14c3a750d9b" containerName="istio-proxy" containerID="cri-o://48469a7175d1eada11ba0d3b73602ff66e162dc95523e5d467ecf837c315a0ba" gracePeriod=30
Oct 29 23:04:16.246413 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.246387    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"]
Oct 29 23:04:16.246599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.246578    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t" podUID="efb3d37c-5657-4075-843c-be4d358e33f9" containerName="istio-proxy" containerID="cri-o://3fee0c0055d2602dce74c528b08d3a6f61ee4283e70d57417fde574852eaab8b" gracePeriod=30
Oct 29 23:04:16.289765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.289739    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8e1b51-65ba-47f7-b7f5-df5fb69a0203-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w\" (UID: \"7b8e1b51-65ba-47f7-b7f5-df5fb69a0203\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w"
Oct 29 23:04:16.289942 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.289848    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkmc5\" (UniqueName: \"kubernetes.io/projected/7b8e1b51-65ba-47f7-b7f5-df5fb69a0203-kube-api-access-vkmc5\") pod \"payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w\" (UID: \"7b8e1b51-65ba-47f7-b7f5-df5fb69a0203\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w"
Oct 29 23:04:16.289942 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.289929    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j48cc\" (UniqueName: \"kubernetes.io/projected/aca44441-e031-4a00-8814-6b5e193b0f4a-kube-api-access-j48cc\") pod \"payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz\" (UID: \"aca44441-e031-4a00-8814-6b5e193b0f4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz"
Oct 29 23:04:16.290167 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.290143    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca44441-e031-4a00-8814-6b5e193b0f4a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz\" (UID: \"aca44441-e031-4a00-8814-6b5e193b0f4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz"
Oct 29 23:04:16.290819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.290761    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8e1b51-65ba-47f7-b7f5-df5fb69a0203-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w\" (UID: \"7b8e1b51-65ba-47f7-b7f5-df5fb69a0203\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w"
Oct 29 23:04:16.290945 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.290910    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca44441-e031-4a00-8814-6b5e193b0f4a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz\" (UID: \"aca44441-e031-4a00-8814-6b5e193b0f4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz"
Oct 29 23:04:16.293696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.293671    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:16.335658 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.335610    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j48cc\" (UniqueName: \"kubernetes.io/projected/aca44441-e031-4a00-8814-6b5e193b0f4a-kube-api-access-j48cc\") pod \"payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz\" (UID: \"aca44441-e031-4a00-8814-6b5e193b0f4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz"
Oct 29 23:04:16.374386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.374354    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkmc5\" (UniqueName: \"kubernetes.io/projected/7b8e1b51-65ba-47f7-b7f5-df5fb69a0203-kube-api-access-vkmc5\") pod \"payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w\" (UID: \"7b8e1b51-65ba-47f7-b7f5-df5fb69a0203\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w"
Oct 29 23:04:16.409737 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.409412    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz"
Oct 29 23:04:16.487103 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.480611    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w"
Oct 29 23:04:16.809334 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.809303    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz"]
Oct 29 23:04:16.809810 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:04:16.809772    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca44441_e031_4a00_8814_6b5e193b0f4a.slice/crio-08fd6fdb490b727d8f7f3e22d8ee06682673635bde57c9e2426e77b51d8b299a WatchSource:0}: Error finding container 08fd6fdb490b727d8f7f3e22d8ee06682673635bde57c9e2426e77b51d8b299a: Status 404 returned error can't find the container with id 08fd6fdb490b727d8f7f3e22d8ee06682673635bde57c9e2426e77b51d8b299a
Oct 29 23:04:16.844235 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:16.844211    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w"]
Oct 29 23:04:16.846141 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:04:16.846108    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8e1b51_65ba_47f7_b7f5_df5fb69a0203.slice/crio-83b8af6bac47b40ebf60b0e79824967c43796f09d903dcc239c94965706653b6 WatchSource:0}: Error finding container 83b8af6bac47b40ebf60b0e79824967c43796f09d903dcc239c94965706653b6: Status 404 returned error can't find the container with id 83b8af6bac47b40ebf60b0e79824967c43796f09d903dcc239c94965706653b6
Oct 29 23:04:17.304059 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:17.304020    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:17.324535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:17.324489    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w" event={"ID":"7b8e1b51-65ba-47f7-b7f5-df5fb69a0203","Type":"ContainerStarted","Data":"a49df011d0b166a6844b6893ad41cec7cb85a8362d63d6b5c4782a6137b87198"}
Oct 29 23:04:17.324696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:17.324585    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w" event={"ID":"7b8e1b51-65ba-47f7-b7f5-df5fb69a0203","Type":"ContainerStarted","Data":"83b8af6bac47b40ebf60b0e79824967c43796f09d903dcc239c94965706653b6"}
Oct 29 23:04:17.324696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:17.324617    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w"
Oct 29 23:04:17.326979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:17.326950    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz" event={"ID":"aca44441-e031-4a00-8814-6b5e193b0f4a","Type":"ContainerStarted","Data":"ebede9e5286498fc132dcf59e66f0e5235f3d06db0c5f6ad99ac128838a494ec"}
Oct 29 23:04:17.327135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:17.327003    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz" event={"ID":"aca44441-e031-4a00-8814-6b5e193b0f4a","Type":"ContainerStarted","Data":"08fd6fdb490b727d8f7f3e22d8ee06682673635bde57c9e2426e77b51d8b299a"}
Oct 29 23:04:17.327135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:17.327030    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz"
Oct 29 23:04:17.370170 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:04:17.370136    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:04:17.371145 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:17.371104    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w" podStartSLOduration=1.3710912720000001 podStartE2EDuration="1.371091272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:17.366995009 +0000 UTC m=+1260.709943754" watchObservedRunningTime="2026-10-29 23:04:17.371091272 +0000 UTC m=+1260.714040016"
Oct 29 23:04:17.412135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:17.412070    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz" podStartSLOduration=1.412056134 podStartE2EDuration="1.412056134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:17.411399292 +0000 UTC m=+1260.754348038" watchObservedRunningTime="2026-10-29 23:04:17.412056134 +0000 UTC m=+1260.755004879"
Oct 29 23:04:17.428295 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:17.428273    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:17.553525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:17.553477    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:19.620138 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:19.620101    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:20.307474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:20.307445    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"
Oct 29 23:04:21.366502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.366474    5428 generic.go:356] "Generic (PLEG): container finished" podID="9eeb8c61-ed7d-420a-b256-e14c3a750d9b" containerID="48469a7175d1eada11ba0d3b73602ff66e162dc95523e5d467ecf837c315a0ba" exitCode=0
Oct 29 23:04:21.366971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.366540    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752" event={"ID":"9eeb8c61-ed7d-420a-b256-e14c3a750d9b","Type":"ContainerDied","Data":"48469a7175d1eada11ba0d3b73602ff66e162dc95523e5d467ecf837c315a0ba"}
Oct 29 23:04:21.366971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.366636    5428 scope.go:122] "RemoveContainer" containerID="6608e117ee2250f826cadcc1a42e0eb48748a583a001124ba960d7039eabf579"
Oct 29 23:04:21.369305 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.369278    5428 generic.go:356] "Generic (PLEG): container finished" podID="a8f4481b-afbf-41b9-b739-df14b6a3e77f" containerID="25bc92ffd492b08ef1b794e443d4c8214242944a2692f6a63fe05a7d5791565e" exitCode=0
Oct 29 23:04:21.369396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.369342    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq" event={"ID":"a8f4481b-afbf-41b9-b739-df14b6a3e77f","Type":"ContainerDied","Data":"25bc92ffd492b08ef1b794e443d4c8214242944a2692f6a63fe05a7d5791565e"}
Oct 29 23:04:21.558860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.558833    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 23:04:21.577788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.577765    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:04:21.710209 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.710163    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istiod-ca-cert\") pod \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") "
Oct 29 23:04:21.710401 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.710238    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-kube-api-access-9vfwv\" (UniqueName: \"kubernetes.io/projected/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-kube-api-access-9vfwv\") pod \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") "
Oct 29 23:04:21.710401 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.710283    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-istio-envoy\") pod \"efb3d37c-5657-4075-843c-be4d358e33f9\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") "
Oct 29 23:04:21.710401 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.710315    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-istio-data\") pod \"efb3d37c-5657-4075-843c-be4d358e33f9\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") "
Oct 29 23:04:21.710401 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.710349    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-workload-socket\") pod \"efb3d37c-5657-4075-843c-be4d358e33f9\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") "
Oct 29 23:04:21.710401 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.710373    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-workload-socket\") pod \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") "
Oct 29 23:04:21.710819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.710459    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb3d37c-5657-4075-843c-be4d358e33f9-kube-api-access-vg85s\" (UniqueName: \"kubernetes.io/projected/efb3d37c-5657-4075-843c-be4d358e33f9-kube-api-access-vg85s\") pod \"efb3d37c-5657-4075-843c-be4d358e33f9\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") "
Oct 29 23:04:21.710819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.710545    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-credential-socket\") pod \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") "
Oct 29 23:04:21.710819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.710583    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-credential-socket\") pod \"efb3d37c-5657-4075-843c-be4d358e33f9\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") "
Oct 29 23:04:21.710819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.710692    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-envoy\") pod \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") "
Oct 29 23:04:21.710819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.710755    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efb3d37c-5657-4075-843c-be4d358e33f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb3d37c-5657-4075-843c-be4d358e33f9-istiod-ca-cert\") pod \"efb3d37c-5657-4075-843c-be4d358e33f9\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") "
Oct 29 23:04:21.711089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.710839    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efb3d37c-5657-4075-843c-be4d358e33f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb3d37c-5657-4075-843c-be4d358e33f9-istio-podinfo\") pod \"efb3d37c-5657-4075-843c-be4d358e33f9\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") "
Oct 29 23:04:21.711089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.710927    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-data\") pod \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") "
Oct 29 23:04:21.711089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.710952    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-workload-certs\") pod \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") "
Oct 29 23:04:21.711089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.711045    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-token\" (UniqueName: \"kubernetes.io/projected/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-token\") pod \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") "
Oct 29 23:04:21.711089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.711081    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-podinfo\") pod \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\" (UID: \"9eeb8c61-ed7d-420a-b256-e14c3a750d9b\") "
Oct 29 23:04:21.711241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.711117    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb3d37c-5657-4075-843c-be4d358e33f9-istio-token\" (UniqueName: \"kubernetes.io/projected/efb3d37c-5657-4075-843c-be4d358e33f9-istio-token\") pod \"efb3d37c-5657-4075-843c-be4d358e33f9\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") "
Oct 29 23:04:21.711241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.711152    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-workload-certs\") pod \"efb3d37c-5657-4075-843c-be4d358e33f9\" (UID: \"efb3d37c-5657-4075-843c-be4d358e33f9\") "
Oct 29 23:04:21.712633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.711500    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-istio-data" pod "efb3d37c-5657-4075-843c-be4d358e33f9" (UID: "efb3d37c-5657-4075-843c-be4d358e33f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:21.712633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.711722    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-workload-certs" pod "efb3d37c-5657-4075-843c-be4d358e33f9" (UID: "efb3d37c-5657-4075-843c-be4d358e33f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:21.712633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.712005    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:21.712633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.712028    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:21.712633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.712258    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-workload-socket" pod "efb3d37c-5657-4075-843c-be4d358e33f9" (UID: "efb3d37c-5657-4075-843c-be4d358e33f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:21.712633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.712462    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-data" pod "9eeb8c61-ed7d-420a-b256-e14c3a750d9b" (UID: "9eeb8c61-ed7d-420a-b256-e14c3a750d9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:21.713360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.713094    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-credential-socket" pod "efb3d37c-5657-4075-843c-be4d358e33f9" (UID: "efb3d37c-5657-4075-843c-be4d358e33f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:21.713857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.713834    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istiod-ca-cert" pod "9eeb8c61-ed7d-420a-b256-e14c3a750d9b" (UID: "9eeb8c61-ed7d-420a-b256-e14c3a750d9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:21.714168 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.714140    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-podinfo" pod "9eeb8c61-ed7d-420a-b256-e14c3a750d9b" (UID: "9eeb8c61-ed7d-420a-b256-e14c3a750d9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:21.714278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.714202    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-kube-api-access-9vfwv" pod "9eeb8c61-ed7d-420a-b256-e14c3a750d9b" (UID: "9eeb8c61-ed7d-420a-b256-e14c3a750d9b"). InnerVolumeSpecName "kube-api-access-9vfwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:21.714278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.714257    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-credential-socket" pod "9eeb8c61-ed7d-420a-b256-e14c3a750d9b" (UID: "9eeb8c61-ed7d-420a-b256-e14c3a750d9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:21.714483 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.714459    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-workload-certs" pod "9eeb8c61-ed7d-420a-b256-e14c3a750d9b" (UID: "9eeb8c61-ed7d-420a-b256-e14c3a750d9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:21.715211 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.715148    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efb3d37c-5657-4075-843c-be4d358e33f9-istio-podinfo" pod "efb3d37c-5657-4075-843c-be4d358e33f9" (UID: "efb3d37c-5657-4075-843c-be4d358e33f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:21.715319 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.715257    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-istio-envoy" pod "efb3d37c-5657-4075-843c-be4d358e33f9" (UID: "efb3d37c-5657-4075-843c-be4d358e33f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:21.715386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.715330    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb3d37c-5657-4075-843c-be4d358e33f9-istiod-ca-cert" pod "efb3d37c-5657-4075-843c-be4d358e33f9" (UID: "efb3d37c-5657-4075-843c-be4d358e33f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:21.715386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.715349    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-token" pod "9eeb8c61-ed7d-420a-b256-e14c3a750d9b" (UID: "9eeb8c61-ed7d-420a-b256-e14c3a750d9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:21.715499 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.715403    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-workload-socket" pod "9eeb8c61-ed7d-420a-b256-e14c3a750d9b" (UID: "9eeb8c61-ed7d-420a-b256-e14c3a750d9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:21.716250 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.716220    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb3d37c-5657-4075-843c-be4d358e33f9-kube-api-access-vg85s" pod "efb3d37c-5657-4075-843c-be4d358e33f9" (UID: "efb3d37c-5657-4075-843c-be4d358e33f9"). InnerVolumeSpecName "kube-api-access-vg85s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:21.716369 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.716349    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb3d37c-5657-4075-843c-be4d358e33f9-istio-token" pod "efb3d37c-5657-4075-843c-be4d358e33f9" (UID: "efb3d37c-5657-4075-843c-be4d358e33f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:21.716707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.716687    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-envoy" pod "9eeb8c61-ed7d-420a-b256-e14c3a750d9b" (UID: "9eeb8c61-ed7d-420a-b256-e14c3a750d9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:21.805381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.805287    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"]
Oct 29 23:04:21.806360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.806330    5428 cpu_manager.go:414] "containerMap: removing container" podUID="efb3d37c-5657-4075-843c-be4d358e33f9" containerName="istio-proxy"
Oct 29 23:04:21.806360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.806359    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb3d37c-5657-4075-843c-be4d358e33f9" containerName="istio-proxy"
Oct 29 23:04:21.806563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.806387    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9eeb8c61-ed7d-420a-b256-e14c3a750d9b" containerName="istio-proxy"
Oct 29 23:04:21.806563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.806395    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eeb8c61-ed7d-420a-b256-e14c3a750d9b" containerName="istio-proxy"
Oct 29 23:04:21.806563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.806447    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9eeb8c61-ed7d-420a-b256-e14c3a750d9b" containerName="istio-proxy"
Oct 29 23:04:21.806563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.806455    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eeb8c61-ed7d-420a-b256-e14c3a750d9b" containerName="istio-proxy"
Oct 29 23:04:21.806740 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.806677    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb3d37c-5657-4075-843c-be4d358e33f9" containerName="istio-proxy"
Oct 29 23:04:21.806740 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.806694    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eeb8c61-ed7d-420a-b256-e14c3a750d9b" containerName="istio-proxy"
Oct 29 23:04:21.806740 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.806706    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eeb8c61-ed7d-420a-b256-e14c3a750d9b" containerName="istio-proxy"
Oct 29 23:04:21.813422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.813228    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:21.815846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.814652    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:21.815846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.814691    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:21.815846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.814706    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:21.815846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.814720    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:21.815846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.814737    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb3d37c-5657-4075-843c-be4d358e33f9-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:21.815846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.814752    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:21.815846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.814767    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vfwv\" (UniqueName: \"kubernetes.io/projected/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-kube-api-access-9vfwv\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:21.815846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.814783    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:21.815846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.814813    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:21.815846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.814828    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:21.815846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.814844    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg85s\" (UniqueName: \"kubernetes.io/projected/efb3d37c-5657-4075-843c-be4d358e33f9-kube-api-access-vg85s\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:21.815846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.814858    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:21.815846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.814871    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb3d37c-5657-4075-843c-be4d358e33f9-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:21.815846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.814883    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eeb8c61-ed7d-420a-b256-e14c3a750d9b-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:21.815846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.814897    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb3d37c-5657-4075-843c-be4d358e33f9-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:21.815846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.814909    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb3d37c-5657-4075-843c-be4d358e33f9-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:21.840935 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.840854    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"]
Oct 29 23:04:21.916902 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.916859    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:21.917093 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.917058    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz2cz\" (UniqueName: \"kubernetes.io/projected/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-kube-api-access-bz2cz\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:21.917166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.917095    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:21.917224 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.917195    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:21.917396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.917289    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:21.917396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.917374    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:21.917615 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.917421    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:21.917615 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.917484    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:21.917615 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.917540    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:21.988416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:21.988389    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:04:22.000926 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.000894    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:22.018599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.018565    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.018724 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.018637    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.018779 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.018748    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2cz\" (UniqueName: \"kubernetes.io/projected/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-kube-api-access-bz2cz\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.018779 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.018772    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.018889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.018869    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.018975 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.018952    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.019118 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.019027    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.019118 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.019075    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.019246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.019138    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.019312 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.019261    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.019443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.019420    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.019524 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.019476    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.019579 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.019532    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.019656 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.019635    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.021373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.021353    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.021594 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.021573    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.053109 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.053006    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.057320 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.057216    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz2cz\" (UniqueName: \"kubernetes.io/projected/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-kube-api-access-bz2cz\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.120061 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.120023    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-workload-socket\") pod \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") "
Oct 29 23:04:22.120256 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.120089    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-credential-socket\") pod \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") "
Oct 29 23:04:22.120330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.120261    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-data\") pod \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") "
Oct 29 23:04:22.120330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.120304    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-workload-certs\") pod \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") "
Oct 29 23:04:22.120441 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.120358    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f4481b-afbf-41b9-b739-df14b6a3e77f-kube-api-access-snsm7\" (UniqueName: \"kubernetes.io/projected/a8f4481b-afbf-41b9-b739-df14b6a3e77f-kube-api-access-snsm7\") pod \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") "
Oct 29 23:04:22.120441 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.120423    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-podinfo\") pod \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") "
Oct 29 23:04:22.120546 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.120473    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istiod-ca-cert\") pod \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") "
Oct 29 23:04:22.120651 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.120621    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-envoy\") pod \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") "
Oct 29 23:04:22.120821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.120772    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-token\" (UniqueName: \"kubernetes.io/projected/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-token\") pod \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\" (UID: \"a8f4481b-afbf-41b9-b739-df14b6a3e77f\") "
Oct 29 23:04:22.121133 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.121108    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-workload-certs" pod "a8f4481b-afbf-41b9-b739-df14b6a3e77f" (UID: "a8f4481b-afbf-41b9-b739-df14b6a3e77f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:22.121497 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.121280    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-workload-socket" pod "a8f4481b-afbf-41b9-b739-df14b6a3e77f" (UID: "a8f4481b-afbf-41b9-b739-df14b6a3e77f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:22.121497 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.121310    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istiod-ca-cert" pod "a8f4481b-afbf-41b9-b739-df14b6a3e77f" (UID: "a8f4481b-afbf-41b9-b739-df14b6a3e77f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:22.121497 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.121473    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-data" pod "a8f4481b-afbf-41b9-b739-df14b6a3e77f" (UID: "a8f4481b-afbf-41b9-b739-df14b6a3e77f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:22.121736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.121707    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:22.121736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.121735    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:22.121917 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.121748    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:22.121917 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.121760    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:22.122045 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.121928    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-credential-socket" pod "a8f4481b-afbf-41b9-b739-df14b6a3e77f" (UID: "a8f4481b-afbf-41b9-b739-df14b6a3e77f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:22.124102 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.124064    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-envoy" pod "a8f4481b-afbf-41b9-b739-df14b6a3e77f" (UID: "a8f4481b-afbf-41b9-b739-df14b6a3e77f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:22.124102 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.124090    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f4481b-afbf-41b9-b739-df14b6a3e77f-kube-api-access-snsm7" pod "a8f4481b-afbf-41b9-b739-df14b6a3e77f" (UID: "a8f4481b-afbf-41b9-b739-df14b6a3e77f"). InnerVolumeSpecName "kube-api-access-snsm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:22.125109 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.125084    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-token" pod "a8f4481b-afbf-41b9-b739-df14b6a3e77f" (UID: "a8f4481b-afbf-41b9-b739-df14b6a3e77f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:22.125674 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.125646    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-podinfo" pod "a8f4481b-afbf-41b9-b739-df14b6a3e77f" (UID: "a8f4481b-afbf-41b9-b739-df14b6a3e77f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:22.147016 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.146987    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:22.222942 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.222911    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snsm7\" (UniqueName: \"kubernetes.io/projected/a8f4481b-afbf-41b9-b739-df14b6a3e77f-kube-api-access-snsm7\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:22.223066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.222944    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:22.223066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.222959    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:22.223066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.222971    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f4481b-afbf-41b9-b739-df14b6a3e77f-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:22.223066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.222986    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f4481b-afbf-41b9-b739-df14b6a3e77f-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:22.378370 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.378337    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"
Oct 29 23:04:22.378370 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.378364    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752" event={"ID":"9eeb8c61-ed7d-420a-b256-e14c3a750d9b","Type":"ContainerDied","Data":"e4d4d962ec02c9dd9f6bac27b99d7d70f1be9bbe5d762e8ea2cb4df48daaca76"}
Oct 29 23:04:22.378974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.378426    5428 scope.go:122] "RemoveContainer" containerID="48469a7175d1eada11ba0d3b73602ff66e162dc95523e5d467ecf837c315a0ba"
Oct 29 23:04:22.379982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.379953    5428 generic.go:356] "Generic (PLEG): container finished" podID="efb3d37c-5657-4075-843c-be4d358e33f9" containerID="3fee0c0055d2602dce74c528b08d3a6f61ee4283e70d57417fde574852eaab8b" exitCode=0
Oct 29 23:04:22.380059 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.380004    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"
Oct 29 23:04:22.380120 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.380004    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t" event={"ID":"efb3d37c-5657-4075-843c-be4d358e33f9","Type":"ContainerDied","Data":"3fee0c0055d2602dce74c528b08d3a6f61ee4283e70d57417fde574852eaab8b"}
Oct 29 23:04:22.380120 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.380114    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t" event={"ID":"efb3d37c-5657-4075-843c-be4d358e33f9","Type":"ContainerDied","Data":"9444f26e6784f35bc438fd3cfbd69e9e60532671a63197c0c151e70b23537b73"}
Oct 29 23:04:22.382000 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.381974    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"
Oct 29 23:04:22.382000 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.381981    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq" event={"ID":"a8f4481b-afbf-41b9-b739-df14b6a3e77f","Type":"ContainerDied","Data":"ea0d438abb530f6ed1d42f8ea6db5d149c7897c8daf3bb83d7d33d9d71cea23f"}
Oct 29 23:04:22.390764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.390743    5428 scope.go:122] "RemoveContainer" containerID="3fee0c0055d2602dce74c528b08d3a6f61ee4283e70d57417fde574852eaab8b"
Oct 29 23:04:22.401210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.401191    5428 scope.go:122] "RemoveContainer" containerID="3fee0c0055d2602dce74c528b08d3a6f61ee4283e70d57417fde574852eaab8b"
Oct 29 23:04:22.401467 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:04:22.401445    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fee0c0055d2602dce74c528b08d3a6f61ee4283e70d57417fde574852eaab8b\": container with ID starting with 3fee0c0055d2602dce74c528b08d3a6f61ee4283e70d57417fde574852eaab8b not found: ID does not exist" containerID="3fee0c0055d2602dce74c528b08d3a6f61ee4283e70d57417fde574852eaab8b"
Oct 29 23:04:22.401514 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.401475    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fee0c0055d2602dce74c528b08d3a6f61ee4283e70d57417fde574852eaab8b"} err="failed to get container status \"3fee0c0055d2602dce74c528b08d3a6f61ee4283e70d57417fde574852eaab8b\": rpc error: code = NotFound desc = could not find container \"3fee0c0055d2602dce74c528b08d3a6f61ee4283e70d57417fde574852eaab8b\": container with ID starting with 3fee0c0055d2602dce74c528b08d3a6f61ee4283e70d57417fde574852eaab8b not found: ID does not exist"
Oct 29 23:04:22.401514 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.401493    5428 scope.go:122] "RemoveContainer" containerID="25bc92ffd492b08ef1b794e443d4c8214242944a2692f6a63fe05a7d5791565e"
Oct 29 23:04:22.448554 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.448534    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"]
Oct 29 23:04:22.452928 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.452906    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-vhm4t"]
Oct 29 23:04:22.485268 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.485243    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"]
Oct 29 23:04:22.496702 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.496681    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-h2752"]
Oct 29 23:04:22.500610 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:04:22.500584    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9212cb2c_a29c_4889_b4a8_73ac06efcdbb.slice/crio-418cdbcf2bc600b5fe7946ceaafd2424f0708a85de8a61395943d9cd07a773e7 WatchSource:0}: Error finding container 418cdbcf2bc600b5fe7946ceaafd2424f0708a85de8a61395943d9cd07a773e7: Status 404 returned error can't find the container with id 418cdbcf2bc600b5fe7946ceaafd2424f0708a85de8a61395943d9cd07a773e7
Oct 29 23:04:22.501239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.501216    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"]
Oct 29 23:04:22.503860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.503781    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:04:22.503982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.503911    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:04:22.503982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.503958    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:04:22.534652 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.534621    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"]
Oct 29 23:04:22.539003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.538980    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-660540-openshift-default-7bf959676f-2twdq"]
Oct 29 23:04:22.781206 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.781121    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eeb8c61-ed7d-420a-b256-e14c3a750d9b" path="/var/lib/kubelet/pods/9eeb8c61-ed7d-420a-b256-e14c3a750d9b/volumes"
Oct 29 23:04:22.781833 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.781796    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8f4481b-afbf-41b9-b739-df14b6a3e77f" path="/var/lib/kubelet/pods/a8f4481b-afbf-41b9-b739-df14b6a3e77f/volumes"
Oct 29 23:04:22.782361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:22.782346    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efb3d37c-5657-4075-843c-be4d358e33f9" path="/var/lib/kubelet/pods/efb3d37c-5657-4075-843c-be4d358e33f9/volumes"
Oct 29 23:04:23.394339 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:23.394301    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv" event={"ID":"9212cb2c-a29c-4889-b4a8-73ac06efcdbb","Type":"ContainerStarted","Data":"f876765f37b53a543a8d009ce6322640c0e3ae063e412677551849bc01550668"}
Oct 29 23:04:23.394879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:23.394371    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv" event={"ID":"9212cb2c-a29c-4889-b4a8-73ac06efcdbb","Type":"ContainerStarted","Data":"418cdbcf2bc600b5fe7946ceaafd2424f0708a85de8a61395943d9cd07a773e7"}
Oct 29 23:04:23.450329 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:23.450273    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv" podStartSLOduration=2.450259927 podStartE2EDuration="2.450259927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:23.45022768 +0000 UTC m=+1266.793176425" watchObservedRunningTime="2026-10-29 23:04:23.450259927 +0000 UTC m=+1266.793208672"
Oct 29 23:04:24.036694 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:24.036658    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:24.148296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:24.148253    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:24.151881 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:24.151847    5428 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 29 23:04:24.152056 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:24.151927    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv" podUID="9212cb2c-a29c-4889-b4a8-73ac06efcdbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 29 23:04:25.148391 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:25.148350    5428 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 29 23:04:25.148918 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:25.148420    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv" podUID="9212cb2c-a29c-4889-b4a8-73ac06efcdbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 29 23:04:26.120690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:26.120654    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:26.148378 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:26.148345    5428 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 29 23:04:26.148536 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:26.148419    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv" podUID="9212cb2c-a29c-4889-b4a8-73ac06efcdbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 29 23:04:26.553694 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:26.553606    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:26.820146 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:26.820065    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:27.148076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:27.148040    5428 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 29 23:04:27.148250 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:27.148108    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv" podUID="9212cb2c-a29c-4889-b4a8-73ac06efcdbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 29 23:04:28.147694 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:28.147617    5428 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 29 23:04:28.147694 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:28.147665    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv" podUID="9212cb2c-a29c-4889-b4a8-73ac06efcdbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 29 23:04:28.339279 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:28.339242    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz"
Oct 29 23:04:28.339446 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:28.339313    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w"
Oct 29 23:04:28.790724 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:28.790690    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:29.152615 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:29.152586    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:29.153054 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:29.153036    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:29.153737 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:29.153717    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:04:30.345574 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:30.345536    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:30.517893 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:30.517862    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:30.637983 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:30.637944    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:30.774130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:30.774090    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:30.893720 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:30.893638    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:31.014278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:31.014241    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:31.248572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:31.245753    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:31.501754 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:31.501664    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:31.627610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:31.622555    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:31.750929 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:31.750890    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:31.865415 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:31.865383    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:31.920259 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:31.920226    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"]
Oct 29 23:04:31.920462 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:31.920441    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy" containerID="cri-o://bb8993ed4e82ae873ea239d84ba5b436c9339c3f3a3ba45237acb9f59d03aac2" gracePeriod=30
Oct 29 23:04:31.932360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:31.932334    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"]
Oct 29 23:04:31.932590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:31.932567    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n" podUID="7c5d481f-83bd-48f3-8945-cd88d304e8d2" containerName="istio-proxy" containerID="cri-o://6a95b91343d39a5bc231348e910b4c2bdcf3cf1e8aa780addaf1ed4594ab4c47" gracePeriod=30
Oct 29 23:04:32.272981 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:32.272898    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:32.460975 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:32.460936    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:32.644716 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:32.644681    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:32.951121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:32.951040    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:33.296950 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:33.296862    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:34.316327 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:34.316288    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:04:34.316785 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:34.316348    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:04:34.866346 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:34.866307    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:36.723403 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.723354    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:36.826752 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.826717    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"]
Oct 29 23:04:36.827628 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.827606    5428 cpu_manager.go:414] "containerMap: removing container" podUID="a8f4481b-afbf-41b9-b739-df14b6a3e77f" containerName="istio-proxy"
Oct 29 23:04:36.827628 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.827628    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f4481b-afbf-41b9-b739-df14b6a3e77f" containerName="istio-proxy"
Oct 29 23:04:36.827880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.827794    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f4481b-afbf-41b9-b739-df14b6a3e77f" containerName="istio-proxy"
Oct 29 23:04:36.831797 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.831776    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.878373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.878336    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"]
Oct 29 23:04:36.896611 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.896573    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.896774 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.896631    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.896774 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.896720    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.896774 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.896762    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.896907 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.896789    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.896907 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.896851    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.896907 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.896880    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.896907 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.896903    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.897088 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.897019    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85bpr\" (UniqueName: \"kubernetes.io/projected/84fa6583-b117-4e41-9ca4-2d5d3742bff9-kube-api-access-85bpr\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.998933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.998853    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.998933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.998904    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.998933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.998926    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.999195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.999049    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85bpr\" (UniqueName: \"kubernetes.io/projected/84fa6583-b117-4e41-9ca4-2d5d3742bff9-kube-api-access-85bpr\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.999195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.999108    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.999195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.999146    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.999367 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.999251    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.999367 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.999314    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.999367 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.999346    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.999521 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.999438    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-data\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.999653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.999624    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-workload-certs\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.999653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.999645    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-credential-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.999783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.999703    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-workload-socket\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:36.999783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:36.999740    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istiod-ca-cert\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:37.001595 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.001566    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-envoy\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:37.001738 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.001671    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-podinfo\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:37.022844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.022816    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-token\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:37.022981 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.022897    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85bpr\" (UniqueName: \"kubernetes.io/projected/84fa6583-b117-4e41-9ca4-2d5d3742bff9-kube-api-access-85bpr\") pod \"e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") " pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:37.146666 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.146634    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:37.215907 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.215873    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 23:04:37.287167 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.287127    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 23:04:37.305401 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.305367    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-workload-socket\") pod \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") "
Oct 29 23:04:37.305580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.305421    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istiod-ca-cert\") pod \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") "
Oct 29 23:04:37.305580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.305464    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c5d481f-83bd-48f3-8945-cd88d304e8d2-kube-api-access-kct9b\" (UniqueName: \"kubernetes.io/projected/7c5d481f-83bd-48f3-8945-cd88d304e8d2-kube-api-access-kct9b\") pod \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") "
Oct 29 23:04:37.305580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.305534    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-podinfo\") pod \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") "
Oct 29 23:04:37.305855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.305594    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-credential-socket\") pod \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") "
Oct 29 23:04:37.305855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.305692    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-workload-socket" pod "7c5d481f-83bd-48f3-8945-cd88d304e8d2" (UID: "7c5d481f-83bd-48f3-8945-cd88d304e8d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:37.305855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.305747    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-envoy\") pod \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") "
Oct 29 23:04:37.305855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.305789    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-data\") pod \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") "
Oct 29 23:04:37.306073 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.305900    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-workload-certs\") pod \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") "
Oct 29 23:04:37.306073 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.305994    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-token\" (UniqueName: \"kubernetes.io/projected/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-token\") pod \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\" (UID: \"7c5d481f-83bd-48f3-8945-cd88d304e8d2\") "
Oct 29 23:04:37.306259 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.306117    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-credential-socket" pod "7c5d481f-83bd-48f3-8945-cd88d304e8d2" (UID: "7c5d481f-83bd-48f3-8945-cd88d304e8d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:37.306432 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.306350    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-data" pod "7c5d481f-83bd-48f3-8945-cd88d304e8d2" (UID: "7c5d481f-83bd-48f3-8945-cd88d304e8d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:37.306695 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.306662    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istiod-ca-cert" pod "7c5d481f-83bd-48f3-8945-cd88d304e8d2" (UID: "7c5d481f-83bd-48f3-8945-cd88d304e8d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:37.306870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.306852    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.306966 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.306875    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.306966 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.306889    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.306966 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.306903    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.307297 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.307249    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-workload-certs" pod "7c5d481f-83bd-48f3-8945-cd88d304e8d2" (UID: "7c5d481f-83bd-48f3-8945-cd88d304e8d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:37.309301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.309257    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5d481f-83bd-48f3-8945-cd88d304e8d2-kube-api-access-kct9b" pod "7c5d481f-83bd-48f3-8945-cd88d304e8d2" (UID: "7c5d481f-83bd-48f3-8945-cd88d304e8d2"). InnerVolumeSpecName "kube-api-access-kct9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:37.309539 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.309491    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-envoy" pod "7c5d481f-83bd-48f3-8945-cd88d304e8d2" (UID: "7c5d481f-83bd-48f3-8945-cd88d304e8d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:37.309893 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.309869    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-token" pod "7c5d481f-83bd-48f3-8945-cd88d304e8d2" (UID: "7c5d481f-83bd-48f3-8945-cd88d304e8d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:37.311056 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.311029    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-podinfo" pod "7c5d481f-83bd-48f3-8945-cd88d304e8d2" (UID: "7c5d481f-83bd-48f3-8945-cd88d304e8d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:37.407560 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.407518    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-envoy\") pod \"9cf99859-eb40-4c1f-a856-c92a700e01eb\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") "
Oct 29 23:04:37.407754 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.407630    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cf99859-eb40-4c1f-a856-c92a700e01eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cf99859-eb40-4c1f-a856-c92a700e01eb-istiod-ca-cert\") pod \"9cf99859-eb40-4c1f-a856-c92a700e01eb\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") "
Oct 29 23:04:37.407882 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.407749    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf99859-eb40-4c1f-a856-c92a700e01eb-kube-api-access-r9j5w\" (UniqueName: \"kubernetes.io/projected/9cf99859-eb40-4c1f-a856-c92a700e01eb-kube-api-access-r9j5w\") pod \"9cf99859-eb40-4c1f-a856-c92a700e01eb\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") "
Oct 29 23:04:37.407882 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.407863    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-credential-socket\") pod \"9cf99859-eb40-4c1f-a856-c92a700e01eb\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") "
Oct 29 23:04:37.407995 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.407977    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-workload-certs\") pod \"9cf99859-eb40-4c1f-a856-c92a700e01eb\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") "
Oct 29 23:04:37.408052 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.408031    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-podinfo\") pod \"9cf99859-eb40-4c1f-a856-c92a700e01eb\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") "
Oct 29 23:04:37.408110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.408037    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cf99859-eb40-4c1f-a856-c92a700e01eb-istiod-ca-cert" pod "9cf99859-eb40-4c1f-a856-c92a700e01eb" (UID: "9cf99859-eb40-4c1f-a856-c92a700e01eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:37.408110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.408065    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-data\") pod \"9cf99859-eb40-4c1f-a856-c92a700e01eb\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") "
Oct 29 23:04:37.408206 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.408120    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-credential-socket" pod "9cf99859-eb40-4c1f-a856-c92a700e01eb" (UID: "9cf99859-eb40-4c1f-a856-c92a700e01eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:37.408206 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.408189    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-token\" (UniqueName: \"kubernetes.io/projected/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-token\") pod \"9cf99859-eb40-4c1f-a856-c92a700e01eb\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") "
Oct 29 23:04:37.408319 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.408236    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-workload-socket\") pod \"9cf99859-eb40-4c1f-a856-c92a700e01eb\" (UID: \"9cf99859-eb40-4c1f-a856-c92a700e01eb\") "
Oct 29 23:04:37.408547 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.408234    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-workload-certs" pod "9cf99859-eb40-4c1f-a856-c92a700e01eb" (UID: "9cf99859-eb40-4c1f-a856-c92a700e01eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:37.408547 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.408422    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-data" pod "9cf99859-eb40-4c1f-a856-c92a700e01eb" (UID: "9cf99859-eb40-4c1f-a856-c92a700e01eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:37.408547 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.408431    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-workload-socket" pod "9cf99859-eb40-4c1f-a856-c92a700e01eb" (UID: "9cf99859-eb40-4c1f-a856-c92a700e01eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:37.408963 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.408941    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kct9b\" (UniqueName: \"kubernetes.io/projected/7c5d481f-83bd-48f3-8945-cd88d304e8d2-kube-api-access-kct9b\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.409063 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.408967    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.409063 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.408980    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cf99859-eb40-4c1f-a856-c92a700e01eb-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.409063 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.408993    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.409063 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.409005    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.409063 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.409017    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c5d481f-83bd-48f3-8945-cd88d304e8d2-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.409063 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.409029    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.409063 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.409041    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.409063 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.409052    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c5d481f-83bd-48f3-8945-cd88d304e8d2-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.410043 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.409064    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.410190 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.410167    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-podinfo" pod "9cf99859-eb40-4c1f-a856-c92a700e01eb" (UID: "9cf99859-eb40-4c1f-a856-c92a700e01eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:37.410272 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.410246    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-token" pod "9cf99859-eb40-4c1f-a856-c92a700e01eb" (UID: "9cf99859-eb40-4c1f-a856-c92a700e01eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:37.410324 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.410304    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-envoy" pod "9cf99859-eb40-4c1f-a856-c92a700e01eb" (UID: "9cf99859-eb40-4c1f-a856-c92a700e01eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:37.410364 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.410318    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf99859-eb40-4c1f-a856-c92a700e01eb-kube-api-access-r9j5w" pod "9cf99859-eb40-4c1f-a856-c92a700e01eb" (UID: "9cf99859-eb40-4c1f-a856-c92a700e01eb"). InnerVolumeSpecName "kube-api-access-r9j5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:37.510329 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.510231    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.510329 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.510263    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.510329 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.510273    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cf99859-eb40-4c1f-a856-c92a700e01eb-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.510329 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.510283    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9j5w\" (UniqueName: \"kubernetes.io/projected/9cf99859-eb40-4c1f-a856-c92a700e01eb-kube-api-access-r9j5w\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:37.520383 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.520354    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"]
Oct 29 23:04:37.521188 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.521163    5428 generic.go:356] "Generic (PLEG): container finished" podID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerID="bb8993ed4e82ae873ea239d84ba5b436c9339c3f3a3ba45237acb9f59d03aac2" exitCode=0
Oct 29 23:04:37.521288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.521232    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"
Oct 29 23:04:37.521288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.521253    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" event={"ID":"9cf99859-eb40-4c1f-a856-c92a700e01eb","Type":"ContainerDied","Data":"bb8993ed4e82ae873ea239d84ba5b436c9339c3f3a3ba45237acb9f59d03aac2"}
Oct 29 23:04:37.521353 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.521305    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww" event={"ID":"9cf99859-eb40-4c1f-a856-c92a700e01eb","Type":"ContainerDied","Data":"1c7716a95424b1cc6f1f49db365085b16f978e2de895e1dbab6f276dc55cab14"}
Oct 29 23:04:37.521353 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.521328    5428 scope.go:122] "RemoveContainer" containerID="bb8993ed4e82ae873ea239d84ba5b436c9339c3f3a3ba45237acb9f59d03aac2"
Oct 29 23:04:37.521851 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:04:37.521821    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fa6583_b117_4e41_9ca4_2d5d3742bff9.slice/crio-9831f4089fc0fc345c305b1faf35568068912253342f06581a6e53bb208dbc4c WatchSource:0}: Error finding container 9831f4089fc0fc345c305b1faf35568068912253342f06581a6e53bb208dbc4c: Status 404 returned error can't find the container with id 9831f4089fc0fc345c305b1faf35568068912253342f06581a6e53bb208dbc4c
Oct 29 23:04:37.522994 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.522933    5428 generic.go:356] "Generic (PLEG): container finished" podID="7c5d481f-83bd-48f3-8945-cd88d304e8d2" containerID="6a95b91343d39a5bc231348e910b4c2bdcf3cf1e8aa780addaf1ed4594ab4c47" exitCode=0
Oct 29 23:04:37.523080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.522993    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"
Oct 29 23:04:37.523080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.523004    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n" event={"ID":"7c5d481f-83bd-48f3-8945-cd88d304e8d2","Type":"ContainerDied","Data":"6a95b91343d39a5bc231348e910b4c2bdcf3cf1e8aa780addaf1ed4594ab4c47"}
Oct 29 23:04:37.523080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.523055    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n" event={"ID":"7c5d481f-83bd-48f3-8945-cd88d304e8d2","Type":"ContainerDied","Data":"b730d96884438d517ac12d8d01866f4cab74547f0c1849a4d6cd369a1c4a92aa"}
Oct 29 23:04:37.525375 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.525342    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:04:37.525493 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.525423    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:04:37.525493 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.525465    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:04:37.534043 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.534025    5428 scope.go:122] "RemoveContainer" containerID="bb8993ed4e82ae873ea239d84ba5b436c9339c3f3a3ba45237acb9f59d03aac2"
Oct 29 23:04:37.534413 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:04:37.534385    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb8993ed4e82ae873ea239d84ba5b436c9339c3f3a3ba45237acb9f59d03aac2\": container with ID starting with bb8993ed4e82ae873ea239d84ba5b436c9339c3f3a3ba45237acb9f59d03aac2 not found: ID does not exist" containerID="bb8993ed4e82ae873ea239d84ba5b436c9339c3f3a3ba45237acb9f59d03aac2"
Oct 29 23:04:37.534491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.534425    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb8993ed4e82ae873ea239d84ba5b436c9339c3f3a3ba45237acb9f59d03aac2"} err="failed to get container status \"bb8993ed4e82ae873ea239d84ba5b436c9339c3f3a3ba45237acb9f59d03aac2\": rpc error: code = NotFound desc = could not find container \"bb8993ed4e82ae873ea239d84ba5b436c9339c3f3a3ba45237acb9f59d03aac2\": container with ID starting with bb8993ed4e82ae873ea239d84ba5b436c9339c3f3a3ba45237acb9f59d03aac2 not found: ID does not exist"
Oct 29 23:04:37.534491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.534450    5428 scope.go:122] "RemoveContainer" containerID="6a95b91343d39a5bc231348e910b4c2bdcf3cf1e8aa780addaf1ed4594ab4c47"
Oct 29 23:04:37.545219 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.545197    5428 scope.go:122] "RemoveContainer" containerID="6a95b91343d39a5bc231348e910b4c2bdcf3cf1e8aa780addaf1ed4594ab4c47"
Oct 29 23:04:37.545513 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:04:37.545485    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a95b91343d39a5bc231348e910b4c2bdcf3cf1e8aa780addaf1ed4594ab4c47\": container with ID starting with 6a95b91343d39a5bc231348e910b4c2bdcf3cf1e8aa780addaf1ed4594ab4c47 not found: ID does not exist" containerID="6a95b91343d39a5bc231348e910b4c2bdcf3cf1e8aa780addaf1ed4594ab4c47"
Oct 29 23:04:37.545606 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.545520    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a95b91343d39a5bc231348e910b4c2bdcf3cf1e8aa780addaf1ed4594ab4c47"} err="failed to get container status \"6a95b91343d39a5bc231348e910b4c2bdcf3cf1e8aa780addaf1ed4594ab4c47\": rpc error: code = NotFound desc = could not find container \"6a95b91343d39a5bc231348e910b4c2bdcf3cf1e8aa780addaf1ed4594ab4c47\": container with ID starting with 6a95b91343d39a5bc231348e910b4c2bdcf3cf1e8aa780addaf1ed4594ab4c47 not found: ID does not exist"
Oct 29 23:04:37.576726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.576705    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"]
Oct 29 23:04:37.584059 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.584036    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-cx84n"]
Oct 29 23:04:37.615663 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.615635    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"]
Oct 29 23:04:37.621656 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.621630    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-nrzww"]
Oct 29 23:04:37.997249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:37.997206    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:38.129579 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:38.129541    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:38.507469 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:38.507438    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:38.641195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:38.641147    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf" event={"ID":"84fa6583-b117-4e41-9ca4-2d5d3742bff9","Type":"ContainerStarted","Data":"cc007067da45d18375047bf1b0f5f6398e1cfa3e51688fd25ece7d06e889179d"}
Oct 29 23:04:38.641372 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:38.641222    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf" event={"ID":"84fa6583-b117-4e41-9ca4-2d5d3742bff9","Type":"ContainerStarted","Data":"9831f4089fc0fc345c305b1faf35568068912253342f06581a6e53bb208dbc4c"}
Oct 29 23:04:38.729809 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:38.729743    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf" podStartSLOduration=2.7297279100000003 podStartE2EDuration="2.72972791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:38.715781115 +0000 UTC m=+1282.058729859" watchObservedRunningTime="2026-10-29 23:04:38.72972791 +0000 UTC m=+1282.072676723"
Oct 29 23:04:38.800330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:38.800249    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c5d481f-83bd-48f3-8945-cd88d304e8d2" path="/var/lib/kubelet/pods/7c5d481f-83bd-48f3-8945-cd88d304e8d2/volumes"
Oct 29 23:04:38.800903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:38.800886    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" path="/var/lib/kubelet/pods/9cf99859-eb40-4c1f-a856-c92a700e01eb/volumes"
Oct 29 23:04:39.117557 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:39.117521    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:39.149898 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:39.149858    5428 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 29 23:04:39.150102 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:39.149937    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf" podUID="84fa6583-b117-4e41-9ca4-2d5d3742bff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 29 23:04:39.150467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:39.150431    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:39.531541 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:39.531461    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:39.688655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:39.688621    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:39.896598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:39.896565    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"]
Oct 29 23:04:39.896816 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:39.896780    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt" podUID="cdf8d152-fc6c-441a-89bf-0d02421e74c2" containerName="istio-proxy" containerID="cri-o://1aaa0eb1ed3bb0d26dc0e322a403cbe64e5279960b4cc50cff30dc1110eb18ca" gracePeriod=30
Oct 29 23:04:39.919055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:39.919019    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:40.147960 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:40.147879    5428 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 29 23:04:40.147960 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:40.147941    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf" podUID="84fa6583-b117-4e41-9ca4-2d5d3742bff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 29 23:04:40.406891 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:40.406772    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:40.541293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:40.541262    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:40.787436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:40.787323    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:41.147360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:41.147325    5428 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 29 23:04:41.147528 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:41.147376    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf" podUID="84fa6583-b117-4e41-9ca4-2d5d3742bff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 29 23:04:41.921234 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:41.921200    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:42.147898 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:42.147864    5428 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 29 23:04:42.148080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:42.147920    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf" podUID="84fa6583-b117-4e41-9ca4-2d5d3742bff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 29 23:04:42.353209 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:42.353119    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:42.836481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:42.836449    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:43.099157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:43.099060    5428 patch_prober.go:29] interesting pod/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:04:43.099157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:43.099119    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt" podUID="cdf8d152-fc6c-441a-89bf-0d02421e74c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:04:43.148030 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:43.147993    5428 patch_prober.go:29] interesting pod/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 29 23:04:43.148884 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:43.148049    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf" podUID="84fa6583-b117-4e41-9ca4-2d5d3742bff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 29 23:04:43.518566 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:43.518532    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:44.190526 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:44.190480    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:44.191050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:44.191033    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:44.191695 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:44.191671    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:04:44.363366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:44.363331    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:45.193440 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.193411    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:04:45.287238 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.287193    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istiod-ca-cert\") pod \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") "
Oct 29 23:04:45.287546 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.287525    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-workload-socket\") pod \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") "
Oct 29 23:04:45.287790 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.287773    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-workload-certs\") pod \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") "
Oct 29 23:04:45.288021 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.288004    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf8d152-fc6c-441a-89bf-0d02421e74c2-kube-api-access-mpqkl\" (UniqueName: \"kubernetes.io/projected/cdf8d152-fc6c-441a-89bf-0d02421e74c2-kube-api-access-mpqkl\") pod \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") "
Oct 29 23:04:45.288135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.288121    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-token\" (UniqueName: \"kubernetes.io/projected/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-token\") pod \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") "
Oct 29 23:04:45.288232 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.288219    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-envoy\") pod \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") "
Oct 29 23:04:45.288575 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.288391    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-podinfo\") pod \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") "
Oct 29 23:04:45.288816 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.288782    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-credential-socket\") pod \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") "
Oct 29 23:04:45.289041 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.289023    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-data\") pod \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\" (UID: \"cdf8d152-fc6c-441a-89bf-0d02421e74c2\") "
Oct 29 23:04:45.293473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.293446    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-data" pod "cdf8d152-fc6c-441a-89bf-0d02421e74c2" (UID: "cdf8d152-fc6c-441a-89bf-0d02421e74c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:45.293787 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.293763    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istiod-ca-cert" pod "cdf8d152-fc6c-441a-89bf-0d02421e74c2" (UID: "cdf8d152-fc6c-441a-89bf-0d02421e74c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:45.294038 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.294021    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-workload-socket" pod "cdf8d152-fc6c-441a-89bf-0d02421e74c2" (UID: "cdf8d152-fc6c-441a-89bf-0d02421e74c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:45.294517 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.294494    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-workload-certs" pod "cdf8d152-fc6c-441a-89bf-0d02421e74c2" (UID: "cdf8d152-fc6c-441a-89bf-0d02421e74c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:45.294935 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.294913    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-credential-socket" pod "cdf8d152-fc6c-441a-89bf-0d02421e74c2" (UID: "cdf8d152-fc6c-441a-89bf-0d02421e74c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:45.297961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.297923    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-envoy" pod "cdf8d152-fc6c-441a-89bf-0d02421e74c2" (UID: "cdf8d152-fc6c-441a-89bf-0d02421e74c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:45.298066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.297974    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-token" pod "cdf8d152-fc6c-441a-89bf-0d02421e74c2" (UID: "cdf8d152-fc6c-441a-89bf-0d02421e74c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:45.300188 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.300150    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-podinfo" pod "cdf8d152-fc6c-441a-89bf-0d02421e74c2" (UID: "cdf8d152-fc6c-441a-89bf-0d02421e74c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:45.316077 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.316048    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf8d152-fc6c-441a-89bf-0d02421e74c2-kube-api-access-mpqkl" pod "cdf8d152-fc6c-441a-89bf-0d02421e74c2" (UID: "cdf8d152-fc6c-441a-89bf-0d02421e74c2"). InnerVolumeSpecName "kube-api-access-mpqkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:45.395361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.395324    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpqkl\" (UniqueName: \"kubernetes.io/projected/cdf8d152-fc6c-441a-89bf-0d02421e74c2-kube-api-access-mpqkl\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:45.395361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.395357    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:45.395361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.395372    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:45.395598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.395382    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:45.395598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.395390    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:45.395598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.395399    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:45.395598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.395409    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf8d152-fc6c-441a-89bf-0d02421e74c2-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:45.395598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.395416    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:45.395598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.395425    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf8d152-fc6c-441a-89bf-0d02421e74c2-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:45.725367 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.725329    5428 generic.go:356] "Generic (PLEG): container finished" podID="cdf8d152-fc6c-441a-89bf-0d02421e74c2" containerID="1aaa0eb1ed3bb0d26dc0e322a403cbe64e5279960b4cc50cff30dc1110eb18ca" exitCode=0
Oct 29 23:04:45.725586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.725396    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"
Oct 29 23:04:45.725586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.725433    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt" event={"ID":"cdf8d152-fc6c-441a-89bf-0d02421e74c2","Type":"ContainerDied","Data":"1aaa0eb1ed3bb0d26dc0e322a403cbe64e5279960b4cc50cff30dc1110eb18ca"}
Oct 29 23:04:45.725586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.725534    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt" event={"ID":"cdf8d152-fc6c-441a-89bf-0d02421e74c2","Type":"ContainerDied","Data":"88bbcdf1822dded508210b6fe967ce920076764a2e729953e08fe57794b15bee"}
Oct 29 23:04:45.725586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.725565    5428 scope.go:122] "RemoveContainer" containerID="1aaa0eb1ed3bb0d26dc0e322a403cbe64e5279960b4cc50cff30dc1110eb18ca"
Oct 29 23:04:45.742495 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.742471    5428 scope.go:122] "RemoveContainer" containerID="1aaa0eb1ed3bb0d26dc0e322a403cbe64e5279960b4cc50cff30dc1110eb18ca"
Oct 29 23:04:45.742893 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:04:45.742863    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aaa0eb1ed3bb0d26dc0e322a403cbe64e5279960b4cc50cff30dc1110eb18ca\": container with ID starting with 1aaa0eb1ed3bb0d26dc0e322a403cbe64e5279960b4cc50cff30dc1110eb18ca not found: ID does not exist" containerID="1aaa0eb1ed3bb0d26dc0e322a403cbe64e5279960b4cc50cff30dc1110eb18ca"
Oct 29 23:04:45.743005 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.742903    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aaa0eb1ed3bb0d26dc0e322a403cbe64e5279960b4cc50cff30dc1110eb18ca"} err="failed to get container status \"1aaa0eb1ed3bb0d26dc0e322a403cbe64e5279960b4cc50cff30dc1110eb18ca\": rpc error: code = NotFound desc = could not find container \"1aaa0eb1ed3bb0d26dc0e322a403cbe64e5279960b4cc50cff30dc1110eb18ca\": container with ID starting with 1aaa0eb1ed3bb0d26dc0e322a403cbe64e5279960b4cc50cff30dc1110eb18ca not found: ID does not exist"
Oct 29 23:04:45.906894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.906786    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"]
Oct 29 23:04:45.912982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:45.912949    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ec4f6d-openshift-default-7768f88b85-xgjvt"]
Oct 29 23:04:46.779895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:46.779860    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdf8d152-fc6c-441a-89bf-0d02421e74c2" path="/var/lib/kubelet/pods/cdf8d152-fc6c-441a-89bf-0d02421e74c2/volumes"
Oct 29 23:04:47.059547 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:47.059459    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:47.201727 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:47.201673    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:47.375076 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:04:47.375031    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:04:47.418399 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:47.416441    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:47.639879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:47.638366    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:48.429217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:48.429176    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:49.185714 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:49.185672    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:49.307114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:49.307084    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:49.755692 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:49.755657    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:50.299161 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:50.299123    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2"]
Oct 29 23:04:50.331609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:50.331576    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf"]
Oct 29 23:04:50.455959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:50.455923    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:50.576207 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:50.576119    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:50.706445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:50.706410    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:51.087046 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:51.087013    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:51.611784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:51.611749    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:51.909455 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:51.909422    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:52.023242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:52.023209    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:52.209595 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:52.209517    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:54.980378 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:54.980314    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:55.300183 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.300093    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2" podUID="790e3088-94e6-441f-9a94-515b0dbd77c0" containerName="payload-processing" containerID="cri-o://a91df94e7ae90fc3435474a537e694f273f2593be2944832938ec94a1b576547" gracePeriod=25
Oct 29 23:04:55.331883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.331847    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf" podUID="21bc23a8-4f83-45d3-842d-08ec70fa41c8" containerName="payload-pre-processing" containerID="cri-o://c159f24d2f53cf611464584671b1b08ed4db164f0ec6f883e5d1cc0381e0c9ea" gracePeriod=25
Oct 29 23:04:55.633993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.633963    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2"
Oct 29 23:04:55.668737 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.668715    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf"
Oct 29 23:04:55.701069 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.701034    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790e3088-94e6-441f-9a94-515b0dbd77c0-kube-api-access-t8lmz\" (UniqueName: \"kubernetes.io/projected/790e3088-94e6-441f-9a94-515b0dbd77c0-kube-api-access-t8lmz\") pod \"790e3088-94e6-441f-9a94-515b0dbd77c0\" (UID: \"790e3088-94e6-441f-9a94-515b0dbd77c0\") "
Oct 29 23:04:55.701269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.701095    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/790e3088-94e6-441f-9a94-515b0dbd77c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790e3088-94e6-441f-9a94-515b0dbd77c0-plugins-config-volume\") pod \"790e3088-94e6-441f-9a94-515b0dbd77c0\" (UID: \"790e3088-94e6-441f-9a94-515b0dbd77c0\") "
Oct 29 23:04:55.702059 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.702031    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790e3088-94e6-441f-9a94-515b0dbd77c0-plugins-config-volume" pod "790e3088-94e6-441f-9a94-515b0dbd77c0" (UID: "790e3088-94e6-441f-9a94-515b0dbd77c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:55.706097 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.706065    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790e3088-94e6-441f-9a94-515b0dbd77c0-kube-api-access-t8lmz" pod "790e3088-94e6-441f-9a94-515b0dbd77c0" (UID: "790e3088-94e6-441f-9a94-515b0dbd77c0"). InnerVolumeSpecName "kube-api-access-t8lmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:55.803256 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.803227    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21bc23a8-4f83-45d3-842d-08ec70fa41c8-kube-api-access-j8qd4\" (UniqueName: \"kubernetes.io/projected/21bc23a8-4f83-45d3-842d-08ec70fa41c8-kube-api-access-j8qd4\") pod \"21bc23a8-4f83-45d3-842d-08ec70fa41c8\" (UID: \"21bc23a8-4f83-45d3-842d-08ec70fa41c8\") "
Oct 29 23:04:55.803451 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.803352    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21bc23a8-4f83-45d3-842d-08ec70fa41c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bc23a8-4f83-45d3-842d-08ec70fa41c8-plugins-config-volume\") pod \"21bc23a8-4f83-45d3-842d-08ec70fa41c8\" (UID: \"21bc23a8-4f83-45d3-842d-08ec70fa41c8\") "
Oct 29 23:04:55.803725 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.803701    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21bc23a8-4f83-45d3-842d-08ec70fa41c8-plugins-config-volume" pod "21bc23a8-4f83-45d3-842d-08ec70fa41c8" (UID: "21bc23a8-4f83-45d3-842d-08ec70fa41c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:55.804052 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.804035    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bc23a8-4f83-45d3-842d-08ec70fa41c8-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:55.804100 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.804054    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8lmz\" (UniqueName: \"kubernetes.io/projected/790e3088-94e6-441f-9a94-515b0dbd77c0-kube-api-access-t8lmz\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:55.804100 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.804064    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/790e3088-94e6-441f-9a94-515b0dbd77c0-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:55.805463 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.805441    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21bc23a8-4f83-45d3-842d-08ec70fa41c8-kube-api-access-j8qd4" pod "21bc23a8-4f83-45d3-842d-08ec70fa41c8" (UID: "21bc23a8-4f83-45d3-842d-08ec70fa41c8"). InnerVolumeSpecName "kube-api-access-j8qd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:55.906088 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.906061    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8qd4\" (UniqueName: \"kubernetes.io/projected/21bc23a8-4f83-45d3-842d-08ec70fa41c8-kube-api-access-j8qd4\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:55.947596 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.947548    5428 generic.go:356] "Generic (PLEG): container finished" podID="790e3088-94e6-441f-9a94-515b0dbd77c0" containerID="a91df94e7ae90fc3435474a537e694f273f2593be2944832938ec94a1b576547" exitCode=0
Oct 29 23:04:55.948084 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.947928    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2"
Oct 29 23:04:55.948695 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.948668    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2" event={"ID":"790e3088-94e6-441f-9a94-515b0dbd77c0","Type":"ContainerDied","Data":"a91df94e7ae90fc3435474a537e694f273f2593be2944832938ec94a1b576547"}
Oct 29 23:04:55.948919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.948899    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2" event={"ID":"790e3088-94e6-441f-9a94-515b0dbd77c0","Type":"ContainerDied","Data":"40df54d33ce85da3cba869446ba0c24ae4f2a571ef853359a9f8b1051e7a2ee7"}
Oct 29 23:04:55.949849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.949120    5428 scope.go:122] "RemoveContainer" containerID="a91df94e7ae90fc3435474a537e694f273f2593be2944832938ec94a1b576547"
Oct 29 23:04:55.951302 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.951277    5428 generic.go:356] "Generic (PLEG): container finished" podID="21bc23a8-4f83-45d3-842d-08ec70fa41c8" containerID="c159f24d2f53cf611464584671b1b08ed4db164f0ec6f883e5d1cc0381e0c9ea" exitCode=0
Oct 29 23:04:55.951410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.951334    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf"
Oct 29 23:04:55.951410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.951342    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf" event={"ID":"21bc23a8-4f83-45d3-842d-08ec70fa41c8","Type":"ContainerDied","Data":"c159f24d2f53cf611464584671b1b08ed4db164f0ec6f883e5d1cc0381e0c9ea"}
Oct 29 23:04:55.951410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.951397    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf" event={"ID":"21bc23a8-4f83-45d3-842d-08ec70fa41c8","Type":"ContainerDied","Data":"5ca80997fedacfd354ab7b13bfd8c7f4c7cd9237fe315453b24d1f883136159e"}
Oct 29 23:04:55.965453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.965435    5428 scope.go:122] "RemoveContainer" containerID="a91df94e7ae90fc3435474a537e694f273f2593be2944832938ec94a1b576547"
Oct 29 23:04:55.965792 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:04:55.965770    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a91df94e7ae90fc3435474a537e694f273f2593be2944832938ec94a1b576547\": container with ID starting with a91df94e7ae90fc3435474a537e694f273f2593be2944832938ec94a1b576547 not found: ID does not exist" containerID="a91df94e7ae90fc3435474a537e694f273f2593be2944832938ec94a1b576547"
Oct 29 23:04:55.965930 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.965868    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a91df94e7ae90fc3435474a537e694f273f2593be2944832938ec94a1b576547"} err="failed to get container status \"a91df94e7ae90fc3435474a537e694f273f2593be2944832938ec94a1b576547\": rpc error: code = NotFound desc = could not find container \"a91df94e7ae90fc3435474a537e694f273f2593be2944832938ec94a1b576547\": container with ID starting with a91df94e7ae90fc3435474a537e694f273f2593be2944832938ec94a1b576547 not found: ID does not exist"
Oct 29 23:04:55.965930 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.965898    5428 scope.go:122] "RemoveContainer" containerID="c159f24d2f53cf611464584671b1b08ed4db164f0ec6f883e5d1cc0381e0c9ea"
Oct 29 23:04:55.977977 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.977957    5428 scope.go:122] "RemoveContainer" containerID="c159f24d2f53cf611464584671b1b08ed4db164f0ec6f883e5d1cc0381e0c9ea"
Oct 29 23:04:55.978370 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:04:55.978330    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c159f24d2f53cf611464584671b1b08ed4db164f0ec6f883e5d1cc0381e0c9ea\": container with ID starting with c159f24d2f53cf611464584671b1b08ed4db164f0ec6f883e5d1cc0381e0c9ea not found: ID does not exist" containerID="c159f24d2f53cf611464584671b1b08ed4db164f0ec6f883e5d1cc0381e0c9ea"
Oct 29 23:04:55.978475 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:55.978376    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c159f24d2f53cf611464584671b1b08ed4db164f0ec6f883e5d1cc0381e0c9ea"} err="failed to get container status \"c159f24d2f53cf611464584671b1b08ed4db164f0ec6f883e5d1cc0381e0c9ea\": rpc error: code = NotFound desc = could not find container \"c159f24d2f53cf611464584671b1b08ed4db164f0ec6f883e5d1cc0381e0c9ea\": container with ID starting with c159f24d2f53cf611464584671b1b08ed4db164f0ec6f883e5d1cc0381e0c9ea not found: ID does not exist"
Oct 29 23:04:56.026050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:56.026015    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf"]
Oct 29 23:04:56.030330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:56.030306    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7af1a5-76b4f78655-td5rf"]
Oct 29 23:04:56.061024 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:56.060992    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2"]
Oct 29 23:04:56.068814 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:56.068768    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7af1a5-65d5d848f4-h4sm2"]
Oct 29 23:04:56.783360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:56.783321    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21bc23a8-4f83-45d3-842d-08ec70fa41c8" path="/var/lib/kubelet/pods/21bc23a8-4f83-45d3-842d-08ec70fa41c8/volumes"
Oct 29 23:04:56.783864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:56.783848    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="790e3088-94e6-441f-9a94-515b0dbd77c0" path="/var/lib/kubelet/pods/790e3088-94e6-441f-9a94-515b0dbd77c0/volumes"
Oct 29 23:04:56.787521 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:56.787500    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:56.996028 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:56.995999    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs"]
Oct 29 23:04:56.996213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:56.996187    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs" podUID="1b38f1ba-e4a0-441e-bf0e-aab36c7f62db" containerName="payload-processing" containerID="cri-o://4df674bbdae857d61bc62b3ae6b2cf1a5065fcc5718055e1cfd21f38337e20e2" gracePeriod=30
Oct 29 23:04:57.036409 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.036345    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p"]
Oct 29 23:04:57.036933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.036558    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p" podUID="c35a2e47-ef8f-4c89-a7c2-1a469211e3a3" containerName="payload-pre-processing" containerID="cri-o://99ecf17535366f704b73ce8efff50b3cc4aefc78f20e0e065e0cdd321b20b566" gracePeriod=30
Oct 29 23:04:57.116233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.116198    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:57.238161 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.238122    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:57.372255 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.366550    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs"
Oct 29 23:04:57.436445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.431450    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b38f1ba-e4a0-441e-bf0e-aab36c7f62db-kube-api-access-9khzp\" (UniqueName: \"kubernetes.io/projected/1b38f1ba-e4a0-441e-bf0e-aab36c7f62db-kube-api-access-9khzp\") pod \"1b38f1ba-e4a0-441e-bf0e-aab36c7f62db\" (UID: \"1b38f1ba-e4a0-441e-bf0e-aab36c7f62db\") "
Oct 29 23:04:57.436445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.431745    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b38f1ba-e4a0-441e-bf0e-aab36c7f62db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b38f1ba-e4a0-441e-bf0e-aab36c7f62db-plugins-config-volume\") pod \"1b38f1ba-e4a0-441e-bf0e-aab36c7f62db\" (UID: \"1b38f1ba-e4a0-441e-bf0e-aab36c7f62db\") "
Oct 29 23:04:57.436445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.433066    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b38f1ba-e4a0-441e-bf0e-aab36c7f62db-plugins-config-volume" pod "1b38f1ba-e4a0-441e-bf0e-aab36c7f62db" (UID: "1b38f1ba-e4a0-441e-bf0e-aab36c7f62db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:57.436445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.433663    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b38f1ba-e4a0-441e-bf0e-aab36c7f62db-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:57.436445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.435968    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p"
Oct 29 23:04:57.471837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.466303    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b38f1ba-e4a0-441e-bf0e-aab36c7f62db-kube-api-access-9khzp" pod "1b38f1ba-e4a0-441e-bf0e-aab36c7f62db" (UID: "1b38f1ba-e4a0-441e-bf0e-aab36c7f62db"). InnerVolumeSpecName "kube-api-access-9khzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:57.536394 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.536348    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c35a2e47-ef8f-4c89-a7c2-1a469211e3a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35a2e47-ef8f-4c89-a7c2-1a469211e3a3-plugins-config-volume\") pod \"c35a2e47-ef8f-4c89-a7c2-1a469211e3a3\" (UID: \"c35a2e47-ef8f-4c89-a7c2-1a469211e3a3\") "
Oct 29 23:04:57.536583 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.536503    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35a2e47-ef8f-4c89-a7c2-1a469211e3a3-kube-api-access-d8dnn\" (UniqueName: \"kubernetes.io/projected/c35a2e47-ef8f-4c89-a7c2-1a469211e3a3-kube-api-access-d8dnn\") pod \"c35a2e47-ef8f-4c89-a7c2-1a469211e3a3\" (UID: \"c35a2e47-ef8f-4c89-a7c2-1a469211e3a3\") "
Oct 29 23:04:57.539732 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.537427    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9khzp\" (UniqueName: \"kubernetes.io/projected/1b38f1ba-e4a0-441e-bf0e-aab36c7f62db-kube-api-access-9khzp\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:57.545011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.544980    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35a2e47-ef8f-4c89-a7c2-1a469211e3a3-plugins-config-volume" pod "c35a2e47-ef8f-4c89-a7c2-1a469211e3a3" (UID: "c35a2e47-ef8f-4c89-a7c2-1a469211e3a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:57.549767 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.549739    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35a2e47-ef8f-4c89-a7c2-1a469211e3a3-kube-api-access-d8dnn" pod "c35a2e47-ef8f-4c89-a7c2-1a469211e3a3" (UID: "c35a2e47-ef8f-4c89-a7c2-1a469211e3a3"). InnerVolumeSpecName "kube-api-access-d8dnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:57.642010 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.641931    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35a2e47-ef8f-4c89-a7c2-1a469211e3a3-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:57.642010 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.641970    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8dnn\" (UniqueName: \"kubernetes.io/projected/c35a2e47-ef8f-4c89-a7c2-1a469211e3a3-kube-api-access-d8dnn\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:04:57.972073 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.972030    5428 generic.go:356] "Generic (PLEG): container finished" podID="c35a2e47-ef8f-4c89-a7c2-1a469211e3a3" containerID="99ecf17535366f704b73ce8efff50b3cc4aefc78f20e0e065e0cdd321b20b566" exitCode=0
Oct 29 23:04:57.972245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.972083    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p"
Oct 29 23:04:57.972245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.972115    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p" event={"ID":"c35a2e47-ef8f-4c89-a7c2-1a469211e3a3","Type":"ContainerDied","Data":"99ecf17535366f704b73ce8efff50b3cc4aefc78f20e0e065e0cdd321b20b566"}
Oct 29 23:04:57.972245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.972172    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p" event={"ID":"c35a2e47-ef8f-4c89-a7c2-1a469211e3a3","Type":"ContainerDied","Data":"c006171ffbc304d512b32f38af22f18349353c529e9c4e075a4eda28b357d099"}
Oct 29 23:04:57.972245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.972194    5428 scope.go:122] "RemoveContainer" containerID="99ecf17535366f704b73ce8efff50b3cc4aefc78f20e0e065e0cdd321b20b566"
Oct 29 23:04:57.973724 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.973701    5428 generic.go:356] "Generic (PLEG): container finished" podID="1b38f1ba-e4a0-441e-bf0e-aab36c7f62db" containerID="4df674bbdae857d61bc62b3ae6b2cf1a5065fcc5718055e1cfd21f38337e20e2" exitCode=0
Oct 29 23:04:57.973855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.973735    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs" event={"ID":"1b38f1ba-e4a0-441e-bf0e-aab36c7f62db","Type":"ContainerDied","Data":"4df674bbdae857d61bc62b3ae6b2cf1a5065fcc5718055e1cfd21f38337e20e2"}
Oct 29 23:04:57.973855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.973754    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs"
Oct 29 23:04:57.973855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.973781    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs" event={"ID":"1b38f1ba-e4a0-441e-bf0e-aab36c7f62db","Type":"ContainerDied","Data":"ec2357201805e74d09bee30573c04a3cab31013ac557e722876600b443480ac5"}
Oct 29 23:04:57.976232 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.976203    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:57.985527 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.985507    5428 scope.go:122] "RemoveContainer" containerID="99ecf17535366f704b73ce8efff50b3cc4aefc78f20e0e065e0cdd321b20b566"
Oct 29 23:04:57.985839 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:04:57.985819    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99ecf17535366f704b73ce8efff50b3cc4aefc78f20e0e065e0cdd321b20b566\": container with ID starting with 99ecf17535366f704b73ce8efff50b3cc4aefc78f20e0e065e0cdd321b20b566 not found: ID does not exist" containerID="99ecf17535366f704b73ce8efff50b3cc4aefc78f20e0e065e0cdd321b20b566"
Oct 29 23:04:57.985932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.985853    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ecf17535366f704b73ce8efff50b3cc4aefc78f20e0e065e0cdd321b20b566"} err="failed to get container status \"99ecf17535366f704b73ce8efff50b3cc4aefc78f20e0e065e0cdd321b20b566\": rpc error: code = NotFound desc = could not find container \"99ecf17535366f704b73ce8efff50b3cc4aefc78f20e0e065e0cdd321b20b566\": container with ID starting with 99ecf17535366f704b73ce8efff50b3cc4aefc78f20e0e065e0cdd321b20b566 not found: ID does not exist"
Oct 29 23:04:57.985932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.985877    5428 scope.go:122] "RemoveContainer" containerID="4df674bbdae857d61bc62b3ae6b2cf1a5065fcc5718055e1cfd21f38337e20e2"
Oct 29 23:04:57.995606 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.995584    5428 scope.go:122] "RemoveContainer" containerID="4df674bbdae857d61bc62b3ae6b2cf1a5065fcc5718055e1cfd21f38337e20e2"
Oct 29 23:04:57.995909 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:04:57.995889    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4df674bbdae857d61bc62b3ae6b2cf1a5065fcc5718055e1cfd21f38337e20e2\": container with ID starting with 4df674bbdae857d61bc62b3ae6b2cf1a5065fcc5718055e1cfd21f38337e20e2 not found: ID does not exist" containerID="4df674bbdae857d61bc62b3ae6b2cf1a5065fcc5718055e1cfd21f38337e20e2"
Oct 29 23:04:57.995984 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:57.995915    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4df674bbdae857d61bc62b3ae6b2cf1a5065fcc5718055e1cfd21f38337e20e2"} err="failed to get container status \"4df674bbdae857d61bc62b3ae6b2cf1a5065fcc5718055e1cfd21f38337e20e2\": rpc error: code = NotFound desc = could not find container \"4df674bbdae857d61bc62b3ae6b2cf1a5065fcc5718055e1cfd21f38337e20e2\": container with ID starting with 4df674bbdae857d61bc62b3ae6b2cf1a5065fcc5718055e1cfd21f38337e20e2 not found: ID does not exist"
Oct 29 23:04:58.026175 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:58.026145    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p"]
Oct 29 23:04:58.037409 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:58.037381    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-10b990-5757c8f6f9-cxt2p"]
Oct 29 23:04:58.063186 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:58.063151    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs"]
Oct 29 23:04:58.066928 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:58.066906    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-10b990-64d887b8d6-tvdvs"]
Oct 29 23:04:58.234295 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:58.234219    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:58.375835 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:58.375786    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:58.514946 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:58.514851    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:58.657071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:58.657039    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:58.775401 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:58.775316    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:58.780934 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:58.780908    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b38f1ba-e4a0-441e-bf0e-aab36c7f62db" path="/var/lib/kubelet/pods/1b38f1ba-e4a0-441e-bf0e-aab36c7f62db/volumes"
Oct 29 23:04:58.781472 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:04:58.781451    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c35a2e47-ef8f-4c89-a7c2-1a469211e3a3" path="/var/lib/kubelet/pods/c35a2e47-ef8f-4c89-a7c2-1a469211e3a3/volumes"
Oct 29 23:05:00.211222 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.211184    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:00.341953 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.341921    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:00.383211 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.383179    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j"]
Oct 29 23:05:00.384037 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384014    5428 cpu_manager.go:414] "containerMap: removing container" podUID="790e3088-94e6-441f-9a94-515b0dbd77c0" containerName="payload-processing"
Oct 29 23:05:00.384037 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384035    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790e3088-94e6-441f-9a94-515b0dbd77c0" containerName="payload-processing"
Oct 29 23:05:00.384251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384054    5428 cpu_manager.go:414] "containerMap: removing container" podUID="1b38f1ba-e4a0-441e-bf0e-aab36c7f62db" containerName="payload-processing"
Oct 29 23:05:00.384251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384061    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b38f1ba-e4a0-441e-bf0e-aab36c7f62db" containerName="payload-processing"
Oct 29 23:05:00.384251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384073    5428 cpu_manager.go:414] "containerMap: removing container" podUID="cdf8d152-fc6c-441a-89bf-0d02421e74c2" containerName="istio-proxy"
Oct 29 23:05:00.384251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384081    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdf8d152-fc6c-441a-89bf-0d02421e74c2" containerName="istio-proxy"
Oct 29 23:05:00.384251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384109    5428 cpu_manager.go:414] "containerMap: removing container" podUID="21bc23a8-4f83-45d3-842d-08ec70fa41c8" containerName="payload-pre-processing"
Oct 29 23:05:00.384251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384116    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21bc23a8-4f83-45d3-842d-08ec70fa41c8" containerName="payload-pre-processing"
Oct 29 23:05:00.384251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384126    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy"
Oct 29 23:05:00.384251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384133    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy"
Oct 29 23:05:00.384251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384159    5428 cpu_manager.go:414] "containerMap: removing container" podUID="7c5d481f-83bd-48f3-8945-cd88d304e8d2" containerName="istio-proxy"
Oct 29 23:05:00.384251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384166    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c5d481f-83bd-48f3-8945-cd88d304e8d2" containerName="istio-proxy"
Oct 29 23:05:00.384251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384179    5428 cpu_manager.go:414] "containerMap: removing container" podUID="c35a2e47-ef8f-4c89-a7c2-1a469211e3a3" containerName="payload-pre-processing"
Oct 29 23:05:00.384251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384186    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c35a2e47-ef8f-4c89-a7c2-1a469211e3a3" containerName="payload-pre-processing"
Oct 29 23:05:00.384872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384380    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="790e3088-94e6-441f-9a94-515b0dbd77c0" containerName="payload-processing"
Oct 29 23:05:00.384872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384398    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b38f1ba-e4a0-441e-bf0e-aab36c7f62db" containerName="payload-processing"
Oct 29 23:05:00.384872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384408    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdf8d152-fc6c-441a-89bf-0d02421e74c2" containerName="istio-proxy"
Oct 29 23:05:00.384872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384423    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="21bc23a8-4f83-45d3-842d-08ec70fa41c8" containerName="payload-pre-processing"
Oct 29 23:05:00.384872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384434    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cf99859-eb40-4c1f-a856-c92a700e01eb" containerName="istio-proxy"
Oct 29 23:05:00.384872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384445    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c5d481f-83bd-48f3-8945-cd88d304e8d2" containerName="istio-proxy"
Oct 29 23:05:00.384872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.384455    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c35a2e47-ef8f-4c89-a7c2-1a469211e3a3" containerName="payload-pre-processing"
Oct 29 23:05:00.389899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.389877    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j"
Oct 29 23:05:00.409633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.409602    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j"]
Oct 29 23:05:00.416364 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.416338    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-618d6c\""
Oct 29 23:05:00.512752 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.512661    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af142501-4ba8-4bb2-a7ce-a1672f90beed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j\" (UID: \"af142501-4ba8-4bb2-a7ce-a1672f90beed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j"
Oct 29 23:05:00.512977 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.512874    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn4kf\" (UniqueName: \"kubernetes.io/projected/af142501-4ba8-4bb2-a7ce-a1672f90beed-kube-api-access-nn4kf\") pod \"payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j\" (UID: \"af142501-4ba8-4bb2-a7ce-a1672f90beed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j"
Oct 29 23:05:00.513932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.513906    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-618d6c-dockercfg-5wlcw\""
Oct 29 23:05:00.518248 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.518223    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb"]
Oct 29 23:05:00.524869 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.524843    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb"]
Oct 29 23:05:00.525108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.525014    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb"
Oct 29 23:05:00.617233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.617185    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af142501-4ba8-4bb2-a7ce-a1672f90beed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j\" (UID: \"af142501-4ba8-4bb2-a7ce-a1672f90beed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j"
Oct 29 23:05:00.617477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.617450    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn4kf\" (UniqueName: \"kubernetes.io/projected/af142501-4ba8-4bb2-a7ce-a1672f90beed-kube-api-access-nn4kf\") pod \"payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j\" (UID: \"af142501-4ba8-4bb2-a7ce-a1672f90beed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j"
Oct 29 23:05:00.618764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.618737    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af142501-4ba8-4bb2-a7ce-a1672f90beed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j\" (UID: \"af142501-4ba8-4bb2-a7ce-a1672f90beed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j"
Oct 29 23:05:00.643871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.643167    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn4kf\" (UniqueName: \"kubernetes.io/projected/af142501-4ba8-4bb2-a7ce-a1672f90beed-kube-api-access-nn4kf\") pod \"payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j\" (UID: \"af142501-4ba8-4bb2-a7ce-a1672f90beed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j"
Oct 29 23:05:00.704956 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.704920    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j"
Oct 29 23:05:00.719196 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.719152    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb7c22d-bbfc-4ade-ab22-db401b342897-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb\" (UID: \"fbb7c22d-bbfc-4ade-ab22-db401b342897\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb"
Oct 29 23:05:00.719469 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.719446    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqzzg\" (UniqueName: \"kubernetes.io/projected/fbb7c22d-bbfc-4ade-ab22-db401b342897-kube-api-access-vqzzg\") pod \"payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb\" (UID: \"fbb7c22d-bbfc-4ade-ab22-db401b342897\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb"
Oct 29 23:05:00.821532 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.821492    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb7c22d-bbfc-4ade-ab22-db401b342897-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb\" (UID: \"fbb7c22d-bbfc-4ade-ab22-db401b342897\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb"
Oct 29 23:05:00.821828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.821780    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqzzg\" (UniqueName: \"kubernetes.io/projected/fbb7c22d-bbfc-4ade-ab22-db401b342897-kube-api-access-vqzzg\") pod \"payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb\" (UID: \"fbb7c22d-bbfc-4ade-ab22-db401b342897\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb"
Oct 29 23:05:00.822664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.822644    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb7c22d-bbfc-4ade-ab22-db401b342897-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb\" (UID: \"fbb7c22d-bbfc-4ade-ab22-db401b342897\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb"
Oct 29 23:05:00.823346 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.823303    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:00.845160 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.845134    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqzzg\" (UniqueName: \"kubernetes.io/projected/fbb7c22d-bbfc-4ade-ab22-db401b342897-kube-api-access-vqzzg\") pod \"payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb\" (UID: \"fbb7c22d-bbfc-4ade-ab22-db401b342897\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb"
Oct 29 23:05:00.859726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.859701    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb"
Oct 29 23:05:00.963677 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:00.963645    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:01.039626 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:05:01.039595    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf142501_4ba8_4bb2_a7ce_a1672f90beed.slice/crio-234d148df61ed933e2da1c7c5d027b74194afacd3ed1b3e4436b26dd86194a81 WatchSource:0}: Error finding container 234d148df61ed933e2da1c7c5d027b74194afacd3ed1b3e4436b26dd86194a81: Status 404 returned error can't find the container with id 234d148df61ed933e2da1c7c5d027b74194afacd3ed1b3e4436b26dd86194a81
Oct 29 23:05:01.040926 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:01.040903    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j"]
Oct 29 23:05:01.185167 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:01.184998    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb"]
Oct 29 23:05:01.188413 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:05:01.188385    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb7c22d_bbfc_4ade_ab22_db401b342897.slice/crio-3231f18039b4f0c79547109497abbd21879bc212b9ba2be6db4d8cb0bbef1944 WatchSource:0}: Error finding container 3231f18039b4f0c79547109497abbd21879bc212b9ba2be6db4d8cb0bbef1944: Status 404 returned error can't find the container with id 3231f18039b4f0c79547109497abbd21879bc212b9ba2be6db4d8cb0bbef1944
Oct 29 23:05:01.857094 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:01.857061    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:02.014742 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:02.014691    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb" event={"ID":"fbb7c22d-bbfc-4ade-ab22-db401b342897","Type":"ContainerStarted","Data":"d1c59ea676f1602d80b144bf6ff7bcf89327e8707e0206c8050103ff7604e594"}
Oct 29 23:05:02.015248 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:02.015213    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb" event={"ID":"fbb7c22d-bbfc-4ade-ab22-db401b342897","Type":"ContainerStarted","Data":"3231f18039b4f0c79547109497abbd21879bc212b9ba2be6db4d8cb0bbef1944"}
Oct 29 23:05:02.015356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:02.015264    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb"
Oct 29 23:05:02.019156 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:02.019127    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j" event={"ID":"af142501-4ba8-4bb2-a7ce-a1672f90beed","Type":"ContainerStarted","Data":"2314dfb20525de4cc85088333d6e0617f22d546c95404685afa2ec0125632957"}
Oct 29 23:05:02.019300 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:02.019178    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j" event={"ID":"af142501-4ba8-4bb2-a7ce-a1672f90beed","Type":"ContainerStarted","Data":"234d148df61ed933e2da1c7c5d027b74194afacd3ed1b3e4436b26dd86194a81"}
Oct 29 23:05:02.019300 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:02.019206    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j"
Oct 29 23:05:02.066478 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:02.066429    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb" podStartSLOduration=2.066415139 podStartE2EDuration="2.066415139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:02.063352162 +0000 UTC m=+1305.406300909" watchObservedRunningTime="2026-10-29 23:05:02.066415139 +0000 UTC m=+1305.409363883"
Oct 29 23:05:02.107605 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:02.107512    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j" podStartSLOduration=2.107499313 podStartE2EDuration="2.107499313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:02.103459736 +0000 UTC m=+1305.446408494" watchObservedRunningTime="2026-10-29 23:05:02.107499313 +0000 UTC m=+1305.450448058"
Oct 29 23:05:02.567794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:02.567760    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:03.036795 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:03.036760    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:03.148691 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:03.148654    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:03.515826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:03.514376    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:04.799101 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.799068    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"]
Oct 29 23:05:04.804192 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.804164    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.831944 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.831899    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"]
Oct 29 23:05:04.882392 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.882355    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-podinfo\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.882557 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.882501    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-workload-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.882557 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.882547    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istiod-ca-cert\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.882713 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.882679    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-token\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.882768 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.882711    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvdlq\" (UniqueName: \"kubernetes.io/projected/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-kube-api-access-pvdlq\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.882844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.882825    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-data\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.882930 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.882908    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-credential-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.882976 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.882963    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-workload-certs\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.883105 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.883078    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-envoy\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.899574 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.899552    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b0c72c-openshift-default-dockercfg-twhkp\""
Oct 29 23:05:04.984655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.984614    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-data\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.984853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.984705    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-credential-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.984853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.984758    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-workload-certs\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.984999 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.984884    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-envoy\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.984999 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.984970    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-podinfo\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.985113 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.985038    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-workload-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.985113 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.985077    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istiod-ca-cert\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.985228 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.985107    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-credential-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.985228 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.985108    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-data\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.985347 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.985290    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-workload-certs\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.985864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.985540    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-token\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.985864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.985582    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvdlq\" (UniqueName: \"kubernetes.io/projected/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-kube-api-access-pvdlq\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.985864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.985755    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istiod-ca-cert\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.985864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.985829    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-workload-socket\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.987880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.987857    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-envoy\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:04.988313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:04.988179    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-podinfo\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:05.009017 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.008989    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-token\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:05.009136 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.009018    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvdlq\" (UniqueName: \"kubernetes.io/projected/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-kube-api-access-pvdlq\") pod \"e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") " pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:05.045171 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.045131    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:05.145986 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.145956    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:05.169593 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.169559    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:05.172373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.172265    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"]
Oct 29 23:05:05.172640 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.172507    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr" podUID="c56f0cf7-d334-4139-b135-8558427a0095" containerName="maas-api" containerID="cri-o://f2a76a2dabd0c5eeb8c4f5f1a8a41f9c2d3962bee174f7415c3240e2a855adf6" gracePeriod=30
Oct 29 23:05:05.363248 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.362609    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:05.474742 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.474704    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"
Oct 29 23:05:05.476743 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.476711    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"]
Oct 29 23:05:05.477030 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.477002    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv" podUID="9212cb2c-a29c-4889-b4a8-73ac06efcdbb" containerName="istio-proxy" containerID="cri-o://f876765f37b53a543a8d009ce6322640c0e3ae063e412677551849bc01550668" gracePeriod=30
Oct 29 23:05:05.506889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.506856    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"]
Oct 29 23:05:05.523704 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.523655    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"]
Oct 29 23:05:05.523935 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.523905    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf" podUID="84fa6583-b117-4e41-9ca4-2d5d3742bff9" containerName="istio-proxy" containerID="cri-o://cc007067da45d18375047bf1b0f5f6398e1cfa3e51688fd25ece7d06e889179d" gracePeriod=30
Oct 29 23:05:05.524185 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.524159    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8" podUID="b886f3b9-2571-4566-9c33-4fea56038612" containerName="istio-proxy" containerID="cri-o://252f9e693c00d8c9c2d86ba19253598581bbfd856960d23fe3d9d4694d66b31e" gracePeriod=30
Oct 29 23:05:05.543607 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:05:05.543572    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85de9d4_72c8_4340_ae0d_f9c8735f5a4f.slice/crio-d0fedd6e166c452a3345526df81925bafdf4fd0274f32a174aaa13421f71c197 WatchSource:0}: Error finding container d0fedd6e166c452a3345526df81925bafdf4fd0274f32a174aaa13421f71c197: Status 404 returned error can't find the container with id d0fedd6e166c452a3345526df81925bafdf4fd0274f32a174aaa13421f71c197
Oct 29 23:05:05.543874 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.543851    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"]
Oct 29 23:05:05.546906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.546867    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:05:05.546979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.546947    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:05:05.547014 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.546992    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:05:05.558646 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.558616    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:05.596462 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.596428    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c56f0cf7-d334-4139-b135-8558427a0095-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56f0cf7-d334-4139-b135-8558427a0095-metrics-certs\") pod \"c56f0cf7-d334-4139-b135-8558427a0095\" (UID: \"c56f0cf7-d334-4139-b135-8558427a0095\") "
Oct 29 23:05:05.597001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.596967    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56f0cf7-d334-4139-b135-8558427a0095-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56f0cf7-d334-4139-b135-8558427a0095-openshift-service-ca\") pod \"c56f0cf7-d334-4139-b135-8558427a0095\" (UID: \"c56f0cf7-d334-4139-b135-8558427a0095\") "
Oct 29 23:05:05.597355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.597328    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56f0cf7-d334-4139-b135-8558427a0095-openshift-service-ca" pod "c56f0cf7-d334-4139-b135-8558427a0095" (UID: "c56f0cf7-d334-4139-b135-8558427a0095"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:05.597485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.597464    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c56f0cf7-d334-4139-b135-8558427a0095-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56f0cf7-d334-4139-b135-8558427a0095-maas-api-tls\") pod \"c56f0cf7-d334-4139-b135-8558427a0095\" (UID: \"c56f0cf7-d334-4139-b135-8558427a0095\") "
Oct 29 23:05:05.599117 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.597900    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56f0cf7-d334-4139-b135-8558427a0095-kube-api-access-dsmlf\" (UniqueName: \"kubernetes.io/projected/c56f0cf7-d334-4139-b135-8558427a0095-kube-api-access-dsmlf\") pod \"c56f0cf7-d334-4139-b135-8558427a0095\" (UID: \"c56f0cf7-d334-4139-b135-8558427a0095\") "
Oct 29 23:05:05.599117 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.599026    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56f0cf7-d334-4139-b135-8558427a0095-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:05.600680 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.600652    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56f0cf7-d334-4139-b135-8558427a0095-maas-api-tls" pod "c56f0cf7-d334-4139-b135-8558427a0095" (UID: "c56f0cf7-d334-4139-b135-8558427a0095"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:05.601940 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.601917    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56f0cf7-d334-4139-b135-8558427a0095-kube-api-access-dsmlf" pod "c56f0cf7-d334-4139-b135-8558427a0095" (UID: "c56f0cf7-d334-4139-b135-8558427a0095"). InnerVolumeSpecName "kube-api-access-dsmlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:05.602024 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.601962    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56f0cf7-d334-4139-b135-8558427a0095-metrics-certs" pod "c56f0cf7-d334-4139-b135-8558427a0095" (UID: "c56f0cf7-d334-4139-b135-8558427a0095"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:05.707784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.705871    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56f0cf7-d334-4139-b135-8558427a0095-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:05.707784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.705914    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsmlf\" (UniqueName: \"kubernetes.io/projected/c56f0cf7-d334-4139-b135-8558427a0095-kube-api-access-dsmlf\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:05.707784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:05.705938    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56f0cf7-d334-4139-b135-8558427a0095-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:06.059149 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.059113    5428 generic.go:356] "Generic (PLEG): container finished" podID="c56f0cf7-d334-4139-b135-8558427a0095" containerID="f2a76a2dabd0c5eeb8c4f5f1a8a41f9c2d3962bee174f7415c3240e2a855adf6" exitCode=0
Oct 29 23:05:06.059634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.059193    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"
Oct 29 23:05:06.059634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.059200    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr" event={"ID":"c56f0cf7-d334-4139-b135-8558427a0095","Type":"ContainerDied","Data":"f2a76a2dabd0c5eeb8c4f5f1a8a41f9c2d3962bee174f7415c3240e2a855adf6"}
Oct 29 23:05:06.059634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.059270    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr" event={"ID":"c56f0cf7-d334-4139-b135-8558427a0095","Type":"ContainerDied","Data":"374509db29a35946ad4f4b7cd127d923edb981647591161626da674d3f17d109"}
Oct 29 23:05:06.059634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.059318    5428 scope.go:122] "RemoveContainer" containerID="f2a76a2dabd0c5eeb8c4f5f1a8a41f9c2d3962bee174f7415c3240e2a855adf6"
Oct 29 23:05:06.061772 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.061742    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg" event={"ID":"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f","Type":"ContainerStarted","Data":"d8c8c7b639148250ac1e83d28a0902d41d54d65a1f6a9cf3293c0c37b9aeedd7"}
Oct 29 23:05:06.061912 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.061852    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg" event={"ID":"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f","Type":"ContainerStarted","Data":"d0fedd6e166c452a3345526df81925bafdf4fd0274f32a174aaa13421f71c197"}
Oct 29 23:05:06.077107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.077056    5428 scope.go:122] "RemoveContainer" containerID="f2a76a2dabd0c5eeb8c4f5f1a8a41f9c2d3962bee174f7415c3240e2a855adf6"
Oct 29 23:05:06.077498 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:05:06.077463    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a76a2dabd0c5eeb8c4f5f1a8a41f9c2d3962bee174f7415c3240e2a855adf6\": container with ID starting with f2a76a2dabd0c5eeb8c4f5f1a8a41f9c2d3962bee174f7415c3240e2a855adf6 not found: ID does not exist" containerID="f2a76a2dabd0c5eeb8c4f5f1a8a41f9c2d3962bee174f7415c3240e2a855adf6"
Oct 29 23:05:06.077580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.077505    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a76a2dabd0c5eeb8c4f5f1a8a41f9c2d3962bee174f7415c3240e2a855adf6"} err="failed to get container status \"f2a76a2dabd0c5eeb8c4f5f1a8a41f9c2d3962bee174f7415c3240e2a855adf6\": rpc error: code = NotFound desc = could not find container \"f2a76a2dabd0c5eeb8c4f5f1a8a41f9c2d3962bee174f7415c3240e2a855adf6\": container with ID starting with f2a76a2dabd0c5eeb8c4f5f1a8a41f9c2d3962bee174f7415c3240e2a855adf6 not found: ID does not exist"
Oct 29 23:05:06.089714 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.089683    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:06.146810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.146766    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:06.148927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.148902    5428 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 29 23:05:06.149021 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.148951    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg" podUID="d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 29 23:05:06.186341 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.186272    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg" podStartSLOduration=2.186250042 podStartE2EDuration="2.186250042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:06.130159107 +0000 UTC m=+1309.473107877" watchObservedRunningTime="2026-10-29 23:05:06.186250042 +0000 UTC m=+1309.529198787"
Oct 29 23:05:06.186890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.186861    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"]
Oct 29 23:05:06.202116 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.202087    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7c486c-7965b5bdb5-54hnr"]
Oct 29 23:05:06.241135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.241091    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:06.427885 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.427843    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:06.796412 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.796320    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56f0cf7-d334-4139-b135-8558427a0095" path="/var/lib/kubelet/pods/c56f0cf7-d334-4139-b135-8558427a0095/volumes"
Oct 29 23:05:06.808439 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.808410    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:06.984427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:06.984386    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:07.146624 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:07.146590    5428 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 29 23:05:07.147078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:07.146643    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg" podUID="d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 29 23:05:07.484678 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:07.484494    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:07.926715 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:07.926686    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:08.146796 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:08.146765    5428 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 29 23:05:08.147327 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:08.146927    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg" podUID="d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 29 23:05:08.601542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:08.601514    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:08.840502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:08.840470    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:08.985239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:08.985207    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:09.146581 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:09.146549    5428 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 29 23:05:09.146742 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:09.146616    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg" podUID="d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 29 23:05:10.129577 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.129543    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:10.146575 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.146545    5428 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 29 23:05:10.146717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.146600    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg" podUID="d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 29 23:05:10.572428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.572391    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:10.701875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.701683    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:10.707580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.707536    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56"]
Oct 29 23:05:10.708679 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.708645    5428 cpu_manager.go:414] "containerMap: removing container" podUID="c56f0cf7-d334-4139-b135-8558427a0095" containerName="maas-api"
Oct 29 23:05:10.708679 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.708669    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56f0cf7-d334-4139-b135-8558427a0095" containerName="maas-api"
Oct 29 23:05:10.708985 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.708951    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56f0cf7-d334-4139-b135-8558427a0095" containerName="maas-api"
Oct 29 23:05:10.713705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.713681    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56"
Oct 29 23:05:10.720566 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.720222    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56"]
Oct 29 23:05:10.741163 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.738671    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-57b28e\""
Oct 29 23:05:10.812133 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.812044    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jljbt\" (UniqueName: \"kubernetes.io/projected/c946b973-a30a-432c-8ee0-5ea240c5ee26-kube-api-access-jljbt\") pod \"payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56\" (UID: \"c946b973-a30a-432c-8ee0-5ea240c5ee26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56"
Oct 29 23:05:10.815535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.815501    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c946b973-a30a-432c-8ee0-5ea240c5ee26-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56\" (UID: \"c946b973-a30a-432c-8ee0-5ea240c5ee26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56"
Oct 29 23:05:10.817407 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.816391    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-57b28e-dockercfg-8kz2n\""
Oct 29 23:05:10.842793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.842408    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg"]
Oct 29 23:05:10.874059 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.874028    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg"]
Oct 29 23:05:10.874388 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.874371    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg"
Oct 29 23:05:10.921705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.920675    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jljbt\" (UniqueName: \"kubernetes.io/projected/c946b973-a30a-432c-8ee0-5ea240c5ee26-kube-api-access-jljbt\") pod \"payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56\" (UID: \"c946b973-a30a-432c-8ee0-5ea240c5ee26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56"
Oct 29 23:05:10.921705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.920786    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c946b973-a30a-432c-8ee0-5ea240c5ee26-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56\" (UID: \"c946b973-a30a-432c-8ee0-5ea240c5ee26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56"
Oct 29 23:05:10.921705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.921580    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c946b973-a30a-432c-8ee0-5ea240c5ee26-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56\" (UID: \"c946b973-a30a-432c-8ee0-5ea240c5ee26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56"
Oct 29 23:05:10.950518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:10.950476    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jljbt\" (UniqueName: \"kubernetes.io/projected/c946b973-a30a-432c-8ee0-5ea240c5ee26-kube-api-access-jljbt\") pod \"payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56\" (UID: \"c946b973-a30a-432c-8ee0-5ea240c5ee26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56"
Oct 29 23:05:11.072243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.044848    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82514398-fdc2-46df-b04f-9d5429af45e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg\" (UID: \"82514398-fdc2-46df-b04f-9d5429af45e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg"
Oct 29 23:05:11.072243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.045105    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbghr\" (UniqueName: \"kubernetes.io/projected/82514398-fdc2-46df-b04f-9d5429af45e8-kube-api-access-cbghr\") pod \"payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg\" (UID: \"82514398-fdc2-46df-b04f-9d5429af45e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg"
Oct 29 23:05:11.072243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.062418    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56"
Oct 29 23:05:11.151941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.146091    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:05:11.151941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.147621    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82514398-fdc2-46df-b04f-9d5429af45e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg\" (UID: \"82514398-fdc2-46df-b04f-9d5429af45e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg"
Oct 29 23:05:11.151941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.147671    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbghr\" (UniqueName: \"kubernetes.io/projected/82514398-fdc2-46df-b04f-9d5429af45e8-kube-api-access-cbghr\") pod \"payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg\" (UID: \"82514398-fdc2-46df-b04f-9d5429af45e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg"
Oct 29 23:05:11.151941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.148732    5428 patch_prober.go:29] interesting pod/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 29 23:05:11.151941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.148779    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg" podUID="d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 29 23:05:11.151941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.150071    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82514398-fdc2-46df-b04f-9d5429af45e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg\" (UID: \"82514398-fdc2-46df-b04f-9d5429af45e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg"
Oct 29 23:05:11.230422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.203977    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbghr\" (UniqueName: \"kubernetes.io/projected/82514398-fdc2-46df-b04f-9d5429af45e8-kube-api-access-cbghr\") pod \"payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg\" (UID: \"82514398-fdc2-46df-b04f-9d5429af45e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg"
Oct 29 23:05:11.230422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.208034    5428 generic.go:356] "Generic (PLEG): container finished" podID="9212cb2c-a29c-4889-b4a8-73ac06efcdbb" containerID="f876765f37b53a543a8d009ce6322640c0e3ae063e412677551849bc01550668" exitCode=0
Oct 29 23:05:11.230422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.208297    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv" event={"ID":"9212cb2c-a29c-4889-b4a8-73ac06efcdbb","Type":"ContainerDied","Data":"f876765f37b53a543a8d009ce6322640c0e3ae063e412677551849bc01550668"}
Oct 29 23:05:11.230422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.211954    5428 generic.go:356] "Generic (PLEG): container finished" podID="84fa6583-b117-4e41-9ca4-2d5d3742bff9" containerID="cc007067da45d18375047bf1b0f5f6398e1cfa3e51688fd25ece7d06e889179d" exitCode=0
Oct 29 23:05:11.230422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.212062    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf" event={"ID":"84fa6583-b117-4e41-9ca4-2d5d3742bff9","Type":"ContainerDied","Data":"cc007067da45d18375047bf1b0f5f6398e1cfa3e51688fd25ece7d06e889179d"}
Oct 29 23:05:11.230422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.215554    5428 generic.go:356] "Generic (PLEG): container finished" podID="b886f3b9-2571-4566-9c33-4fea56038612" containerID="252f9e693c00d8c9c2d86ba19253598581bbfd856960d23fe3d9d4694d66b31e" exitCode=0
Oct 29 23:05:11.230422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.215649    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8" event={"ID":"b886f3b9-2571-4566-9c33-4fea56038612","Type":"ContainerDied","Data":"252f9e693c00d8c9c2d86ba19253598581bbfd856960d23fe3d9d4694d66b31e"}
Oct 29 23:05:11.230422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.215696    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8" event={"ID":"b886f3b9-2571-4566-9c33-4fea56038612","Type":"ContainerDied","Data":"07ad30eea6b3652573f1a38790ae3f66aa271718a0caec706fa40606b045a1d2"}
Oct 29 23:05:11.230422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.215728    5428 scope.go:122] "RemoveContainer" containerID="252f9e693c00d8c9c2d86ba19253598581bbfd856960d23fe3d9d4694d66b31e"
Oct 29 23:05:11.230422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.216015    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"
Oct 29 23:05:11.230422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.219323    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg"
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.258944    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-workload-socket\") pod \"b886f3b9-2571-4566-9c33-4fea56038612\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") "
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.259000    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-istio-data\") pod \"b886f3b9-2571-4566-9c33-4fea56038612\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") "
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.259156    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b886f3b9-2571-4566-9c33-4fea56038612-istio-token\" (UniqueName: \"kubernetes.io/projected/b886f3b9-2571-4566-9c33-4fea56038612-istio-token\") pod \"b886f3b9-2571-4566-9c33-4fea56038612\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") "
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.259248    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-workload-certs\") pod \"b886f3b9-2571-4566-9c33-4fea56038612\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") "
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.259350    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-istio-envoy\") pod \"b886f3b9-2571-4566-9c33-4fea56038612\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") "
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.259391    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b886f3b9-2571-4566-9c33-4fea56038612-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b886f3b9-2571-4566-9c33-4fea56038612-istio-podinfo\") pod \"b886f3b9-2571-4566-9c33-4fea56038612\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") "
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.259479    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-credential-socket\") pod \"b886f3b9-2571-4566-9c33-4fea56038612\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") "
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.259507    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b886f3b9-2571-4566-9c33-4fea56038612-kube-api-access-btv2m\" (UniqueName: \"kubernetes.io/projected/b886f3b9-2571-4566-9c33-4fea56038612-kube-api-access-btv2m\") pod \"b886f3b9-2571-4566-9c33-4fea56038612\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") "
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.259584    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b886f3b9-2571-4566-9c33-4fea56038612-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b886f3b9-2571-4566-9c33-4fea56038612-istiod-ca-cert\") pod \"b886f3b9-2571-4566-9c33-4fea56038612\" (UID: \"b886f3b9-2571-4566-9c33-4fea56038612\") "
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.260168    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-workload-socket" pod "b886f3b9-2571-4566-9c33-4fea56038612" (UID: "b886f3b9-2571-4566-9c33-4fea56038612"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.260306    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.260554    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-workload-certs" pod "b886f3b9-2571-4566-9c33-4fea56038612" (UID: "b886f3b9-2571-4566-9c33-4fea56038612"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.260723    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-credential-socket" pod "b886f3b9-2571-4566-9c33-4fea56038612" (UID: "b886f3b9-2571-4566-9c33-4fea56038612"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.261036    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-istio-data" pod "b886f3b9-2571-4566-9c33-4fea56038612" (UID: "b886f3b9-2571-4566-9c33-4fea56038612"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.269772    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b886f3b9-2571-4566-9c33-4fea56038612-kube-api-access-btv2m" pod "b886f3b9-2571-4566-9c33-4fea56038612" (UID: "b886f3b9-2571-4566-9c33-4fea56038612"). InnerVolumeSpecName "kube-api-access-btv2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.269892    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-istio-envoy" pod "b886f3b9-2571-4566-9c33-4fea56038612" (UID: "b886f3b9-2571-4566-9c33-4fea56038612"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.271461    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b886f3b9-2571-4566-9c33-4fea56038612-istio-token" pod "b886f3b9-2571-4566-9c33-4fea56038612" (UID: "b886f3b9-2571-4566-9c33-4fea56038612"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.271830    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b886f3b9-2571-4566-9c33-4fea56038612-istio-podinfo" pod "b886f3b9-2571-4566-9c33-4fea56038612" (UID: "b886f3b9-2571-4566-9c33-4fea56038612"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.281645    5428 scope.go:122] "RemoveContainer" containerID="252f9e693c00d8c9c2d86ba19253598581bbfd856960d23fe3d9d4694d66b31e"
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:05:11.285118    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"252f9e693c00d8c9c2d86ba19253598581bbfd856960d23fe3d9d4694d66b31e\": container with ID starting with 252f9e693c00d8c9c2d86ba19253598581bbfd856960d23fe3d9d4694d66b31e not found: ID does not exist" containerID="252f9e693c00d8c9c2d86ba19253598581bbfd856960d23fe3d9d4694d66b31e"
Oct 29 23:05:11.290845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.285153    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"252f9e693c00d8c9c2d86ba19253598581bbfd856960d23fe3d9d4694d66b31e"} err="failed to get container status \"252f9e693c00d8c9c2d86ba19253598581bbfd856960d23fe3d9d4694d66b31e\": rpc error: code = NotFound desc = could not find container \"252f9e693c00d8c9c2d86ba19253598581bbfd856960d23fe3d9d4694d66b31e\": container with ID starting with 252f9e693c00d8c9c2d86ba19253598581bbfd856960d23fe3d9d4694d66b31e not found: ID does not exist"
Oct 29 23:05:11.293316 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.291760    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b886f3b9-2571-4566-9c33-4fea56038612-istiod-ca-cert" pod "b886f3b9-2571-4566-9c33-4fea56038612" (UID: "b886f3b9-2571-4566-9c33-4fea56038612"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:11.298311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.297923    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:05:11.302140 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.302048    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:05:11.396216 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.396184    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istiod-ca-cert\") pod \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") "
Oct 29 23:05:11.396775 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.396675    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istiod-ca-cert" pod "84fa6583-b117-4e41-9ca4-2d5d3742bff9" (UID: "84fa6583-b117-4e41-9ca4-2d5d3742bff9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:11.397330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.396842    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-workload-socket\") pod \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") "
Oct 29 23:05:11.397330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.397231    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-podinfo\") pod \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") "
Oct 29 23:05:11.397330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.397278    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-workload-socket" pod "9212cb2c-a29c-4889-b4a8-73ac06efcdbb" (UID: "9212cb2c-a29c-4889-b4a8-73ac06efcdbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:11.397954 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.397720    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-podinfo\") pod \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") "
Oct 29 23:05:11.398173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.398152    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-credential-socket\") pod \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") "
Oct 29 23:05:11.398271 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.398206    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-token\" (UniqueName: \"kubernetes.io/projected/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-token\") pod \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") "
Oct 29 23:05:11.398271 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.398233    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-data\") pod \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") "
Oct 29 23:05:11.398271 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.398264    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-data\") pod \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") "
Oct 29 23:05:11.398489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.398322    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84fa6583-b117-4e41-9ca4-2d5d3742bff9-kube-api-access-85bpr\" (UniqueName: \"kubernetes.io/projected/84fa6583-b117-4e41-9ca4-2d5d3742bff9-kube-api-access-85bpr\") pod \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") "
Oct 29 23:05:11.398489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.398388    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-envoy\") pod \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") "
Oct 29 23:05:11.398489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.398416    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-workload-certs\") pod \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") "
Oct 29 23:05:11.398489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.398461    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-credential-socket\") pod \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") "
Oct 29 23:05:11.398703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.398525    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istiod-ca-cert\") pod \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") "
Oct 29 23:05:11.398703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.398567    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-envoy\") pod \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") "
Oct 29 23:05:11.398703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.398635    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-workload-certs\") pod \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") "
Oct 29 23:05:11.398906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.398712    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-workload-socket\") pod \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\" (UID: \"84fa6583-b117-4e41-9ca4-2d5d3742bff9\") "
Oct 29 23:05:11.398959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.398916    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-kube-api-access-bz2cz\" (UniqueName: \"kubernetes.io/projected/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-kube-api-access-bz2cz\") pod \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") "
Oct 29 23:05:11.399013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.398966    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-token\" (UniqueName: \"kubernetes.io/projected/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-token\") pod \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\" (UID: \"9212cb2c-a29c-4889-b4a8-73ac06efcdbb\") "
Oct 29 23:05:11.399066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.399030    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-data" pod "84fa6583-b117-4e41-9ca4-2d5d3742bff9" (UID: "84fa6583-b117-4e41-9ca4-2d5d3742bff9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:11.399269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.399245    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-data" pod "9212cb2c-a29c-4889-b4a8-73ac06efcdbb" (UID: "9212cb2c-a29c-4889-b4a8-73ac06efcdbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:11.399867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.399795    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.400012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.399877    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.400012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.399891    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b886f3b9-2571-4566-9c33-4fea56038612-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.400012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.399906    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.400012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.399921    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.400012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.399935    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btv2m\" (UniqueName: \"kubernetes.io/projected/b886f3b9-2571-4566-9c33-4fea56038612-kube-api-access-btv2m\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.400012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.399950    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.400012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.399964    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b886f3b9-2571-4566-9c33-4fea56038612-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.400012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.399977    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b886f3b9-2571-4566-9c33-4fea56038612-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.400012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.399989    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.400012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.400002    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.400012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.400015    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b886f3b9-2571-4566-9c33-4fea56038612-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.400832 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.400210    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-credential-socket" pod "84fa6583-b117-4e41-9ca4-2d5d3742bff9" (UID: "84fa6583-b117-4e41-9ca4-2d5d3742bff9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:11.401118 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.401089    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-token" pod "84fa6583-b117-4e41-9ca4-2d5d3742bff9" (UID: "84fa6583-b117-4e41-9ca4-2d5d3742bff9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:11.401332 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.401311    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-workload-socket" pod "84fa6583-b117-4e41-9ca4-2d5d3742bff9" (UID: "84fa6583-b117-4e41-9ca4-2d5d3742bff9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:11.402431 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.401504    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-workload-certs" pod "9212cb2c-a29c-4889-b4a8-73ac06efcdbb" (UID: "9212cb2c-a29c-4889-b4a8-73ac06efcdbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:11.402431 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.401700    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-credential-socket" pod "9212cb2c-a29c-4889-b4a8-73ac06efcdbb" (UID: "9212cb2c-a29c-4889-b4a8-73ac06efcdbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:11.402431 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.401884    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-workload-certs" pod "84fa6583-b117-4e41-9ca4-2d5d3742bff9" (UID: "84fa6583-b117-4e41-9ca4-2d5d3742bff9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:11.402431 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.402207    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istiod-ca-cert" pod "9212cb2c-a29c-4889-b4a8-73ac06efcdbb" (UID: "9212cb2c-a29c-4889-b4a8-73ac06efcdbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:11.404842 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.404787    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-kube-api-access-bz2cz" pod "9212cb2c-a29c-4889-b4a8-73ac06efcdbb" (UID: "9212cb2c-a29c-4889-b4a8-73ac06efcdbb"). InnerVolumeSpecName "kube-api-access-bz2cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:11.405074 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.405048    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-token" pod "9212cb2c-a29c-4889-b4a8-73ac06efcdbb" (UID: "9212cb2c-a29c-4889-b4a8-73ac06efcdbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:11.408034 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.406067    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fa6583-b117-4e41-9ca4-2d5d3742bff9-kube-api-access-85bpr" pod "84fa6583-b117-4e41-9ca4-2d5d3742bff9" (UID: "84fa6583-b117-4e41-9ca4-2d5d3742bff9"). InnerVolumeSpecName "kube-api-access-85bpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:11.408034 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.406425    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-podinfo" pod "9212cb2c-a29c-4889-b4a8-73ac06efcdbb" (UID: "9212cb2c-a29c-4889-b4a8-73ac06efcdbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:05:11.408034 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.407766    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-podinfo" pod "84fa6583-b117-4e41-9ca4-2d5d3742bff9" (UID: "84fa6583-b117-4e41-9ca4-2d5d3742bff9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:05:11.408229 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.408161    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-envoy" pod "9212cb2c-a29c-4889-b4a8-73ac06efcdbb" (UID: "9212cb2c-a29c-4889-b4a8-73ac06efcdbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:11.409576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.409546    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-envoy" pod "84fa6583-b117-4e41-9ca4-2d5d3742bff9" (UID: "84fa6583-b117-4e41-9ca4-2d5d3742bff9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:11.505051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.503665    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.505051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.503698    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.505051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.503712    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.505051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.503726    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.505051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.503742    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.505051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.503757    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85bpr\" (UniqueName: \"kubernetes.io/projected/84fa6583-b117-4e41-9ca4-2d5d3742bff9-kube-api-access-85bpr\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.505051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.503771    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.505051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.503784    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.505892 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.505741    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.505892 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.505776    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.505892 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.505794    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.505892 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.505820    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.505892 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.505833    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fa6583-b117-4e41-9ca4-2d5d3742bff9-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.505892 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.505846    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz2cz\" (UniqueName: \"kubernetes.io/projected/9212cb2c-a29c-4889-b4a8-73ac06efcdbb-kube-api-access-bz2cz\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.551383 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.551355    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56"]
Oct 29 23:05:11.619047 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:05:11.619020    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82514398_fdc2_46df_b04f_9d5429af45e8.slice/crio-298ca4c7081e0fe2accf18b55a0350024c9013726f219f0ced5ed4065b546160 WatchSource:0}: Error finding container 298ca4c7081e0fe2accf18b55a0350024c9013726f219f0ced5ed4065b546160: Status 404 returned error can't find the container with id 298ca4c7081e0fe2accf18b55a0350024c9013726f219f0ced5ed4065b546160
Oct 29 23:05:11.619378 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.619353    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg"]
Oct 29 23:05:11.639568 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.639530    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"]
Oct 29 23:05:11.646543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:11.646517    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-5hxp8"]
Oct 29 23:05:12.151591 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.151558    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:12.152122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.152102    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:12.152792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.152775    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:05:12.224636 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.224604    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56" event={"ID":"c946b973-a30a-432c-8ee0-5ea240c5ee26","Type":"ContainerStarted","Data":"78f05a9b7514adec473becb99bb2f44c22e0f8ff1235160dfc72356ca1bb12f1"}
Oct 29 23:05:12.224830 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.224660    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56"
Oct 29 23:05:12.224830 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.224683    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56" event={"ID":"c946b973-a30a-432c-8ee0-5ea240c5ee26","Type":"ContainerStarted","Data":"912303f6693ceb6499e72ecee858049403645f17fbec4f5f736f30a8c4108867"}
Oct 29 23:05:12.226192 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.226163    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"
Oct 29 23:05:12.226192 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.226186    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf" event={"ID":"84fa6583-b117-4e41-9ca4-2d5d3742bff9","Type":"ContainerDied","Data":"9831f4089fc0fc345c305b1faf35568068912253342f06581a6e53bb208dbc4c"}
Oct 29 23:05:12.226390 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.226236    5428 scope.go:122] "RemoveContainer" containerID="cc007067da45d18375047bf1b0f5f6398e1cfa3e51688fd25ece7d06e889179d"
Oct 29 23:05:12.229309 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.229287    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg" event={"ID":"82514398-fdc2-46df-b04f-9d5429af45e8","Type":"ContainerStarted","Data":"3b5433942519c7eb1ff119775c0c4043603841f713af972aea5367f803abe773"}
Oct 29 23:05:12.229423 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.229326    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg" event={"ID":"82514398-fdc2-46df-b04f-9d5429af45e8","Type":"ContainerStarted","Data":"298ca4c7081e0fe2accf18b55a0350024c9013726f219f0ced5ed4065b546160"}
Oct 29 23:05:12.229423 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.229360    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg"
Oct 29 23:05:12.230769 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.230752    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"
Oct 29 23:05:12.230860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.230772    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv" event={"ID":"9212cb2c-a29c-4889-b4a8-73ac06efcdbb","Type":"ContainerDied","Data":"418cdbcf2bc600b5fe7946ceaafd2424f0708a85de8a61395943d9cd07a773e7"}
Oct 29 23:05:12.241447 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.241426    5428 scope.go:122] "RemoveContainer" containerID="f876765f37b53a543a8d009ce6322640c0e3ae063e412677551849bc01550668"
Oct 29 23:05:12.285217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.285150    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg" podStartSLOduration=2.2851281549999998 podStartE2EDuration="2.285128155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:12.279795655 +0000 UTC m=+1315.622744402" watchObservedRunningTime="2026-10-29 23:05:12.285128155 +0000 UTC m=+1315.628076901"
Oct 29 23:05:12.338632 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.338576    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56" podStartSLOduration=2.338561905 podStartE2EDuration="2.338561905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:12.337299396 +0000 UTC m=+1315.680248169" watchObservedRunningTime="2026-10-29 23:05:12.338561905 +0000 UTC m=+1315.681510649"
Oct 29 23:05:12.375121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.375082    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"]
Oct 29 23:05:12.380318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.380291    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-8hsbf"]
Oct 29 23:05:12.415104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.415028    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"]
Oct 29 23:05:12.426907 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.426875    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-10b990-openshift-default-7569b9cb87-gkmsv"]
Oct 29 23:05:12.780314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.780229    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84fa6583-b117-4e41-9ca4-2d5d3742bff9" path="/var/lib/kubelet/pods/84fa6583-b117-4e41-9ca4-2d5d3742bff9/volumes"
Oct 29 23:05:12.780870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.780844    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9212cb2c-a29c-4889-b4a8-73ac06efcdbb" path="/var/lib/kubelet/pods/9212cb2c-a29c-4889-b4a8-73ac06efcdbb/volumes"
Oct 29 23:05:12.781327 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:12.781313    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b886f3b9-2571-4566-9c33-4fea56038612" path="/var/lib/kubelet/pods/b886f3b9-2571-4566-9c33-4fea56038612/volumes"
Oct 29 23:05:13.032657 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:13.032508    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv"]
Oct 29 23:05:13.035234 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:13.035207    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j"
Oct 29 23:05:13.035380 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:13.035276    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb"
Oct 29 23:05:13.037325 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:13.037300    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4372d16f-tmxwv"]
Oct 29 23:05:13.434356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:13.434320    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:13.566902 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:13.566871    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:13.748691 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:13.748611    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:13.870472 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:13.870436    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:14.780339 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:14.780300    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a434fd6-4f8e-4a9d-a7c8-389a718a45f8" path="/var/lib/kubelet/pods/1a434fd6-4f8e-4a9d-a7c8-389a718a45f8/volumes"
Oct 29 23:05:15.595287 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:15.595252    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:16.078136 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:16.078095    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:16.255511 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:16.255479    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:17.050735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:17.050703    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:17.368543 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:05:17.368509    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:05:18.384911 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:18.384867    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:18.501442 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:18.501395    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:19.194252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:19.194220    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:20.163774 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.163735    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:20.270063 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.270017    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2"]
Oct 29 23:05:20.271234 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.271208    5428 cpu_manager.go:414] "containerMap: removing container" podUID="84fa6583-b117-4e41-9ca4-2d5d3742bff9" containerName="istio-proxy"
Oct 29 23:05:20.271234 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.271231    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fa6583-b117-4e41-9ca4-2d5d3742bff9" containerName="istio-proxy"
Oct 29 23:05:20.271369 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.271244    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9212cb2c-a29c-4889-b4a8-73ac06efcdbb" containerName="istio-proxy"
Oct 29 23:05:20.271369 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.271250    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9212cb2c-a29c-4889-b4a8-73ac06efcdbb" containerName="istio-proxy"
Oct 29 23:05:20.271369 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.271291    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b886f3b9-2571-4566-9c33-4fea56038612" containerName="istio-proxy"
Oct 29 23:05:20.271369 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.271297    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b886f3b9-2571-4566-9c33-4fea56038612" containerName="istio-proxy"
Oct 29 23:05:20.271492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.271397    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fa6583-b117-4e41-9ca4-2d5d3742bff9" containerName="istio-proxy"
Oct 29 23:05:20.271492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.271404    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9212cb2c-a29c-4889-b4a8-73ac06efcdbb" containerName="istio-proxy"
Oct 29 23:05:20.271492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.271420    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b886f3b9-2571-4566-9c33-4fea56038612" containerName="istio-proxy"
Oct 29 23:05:20.276775 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.276758    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2"
Oct 29 23:05:20.290114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.290087    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:20.293842 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.293819    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2"]
Oct 29 23:05:20.293977 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.293951    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ce5978\""
Oct 29 23:05:20.348081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.348042    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62ks\" (UniqueName: \"kubernetes.io/projected/44b8ab25-5d40-45fc-8a55-9ce13bcf8646-kube-api-access-s62ks\") pod \"payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2\" (UID: \"44b8ab25-5d40-45fc-8a55-9ce13bcf8646\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2"
Oct 29 23:05:20.348249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.348094    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b8ab25-5d40-45fc-8a55-9ce13bcf8646-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2\" (UID: \"44b8ab25-5d40-45fc-8a55-9ce13bcf8646\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2"
Oct 29 23:05:20.450294 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.449880    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62ks\" (UniqueName: \"kubernetes.io/projected/44b8ab25-5d40-45fc-8a55-9ce13bcf8646-kube-api-access-s62ks\") pod \"payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2\" (UID: \"44b8ab25-5d40-45fc-8a55-9ce13bcf8646\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2"
Oct 29 23:05:20.450294 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.449952    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b8ab25-5d40-45fc-8a55-9ce13bcf8646-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2\" (UID: \"44b8ab25-5d40-45fc-8a55-9ce13bcf8646\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2"
Oct 29 23:05:20.451600 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.451571    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b8ab25-5d40-45fc-8a55-9ce13bcf8646-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2\" (UID: \"44b8ab25-5d40-45fc-8a55-9ce13bcf8646\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2"
Oct 29 23:05:20.474655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.474627    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62ks\" (UniqueName: \"kubernetes.io/projected/44b8ab25-5d40-45fc-8a55-9ce13bcf8646-kube-api-access-s62ks\") pod \"payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2\" (UID: \"44b8ab25-5d40-45fc-8a55-9ce13bcf8646\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2"
Oct 29 23:05:20.595445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.595417    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2"
Oct 29 23:05:20.875750 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.875715    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:20.926138 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.926063    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2"]
Oct 29 23:05:20.928353 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:05:20.928326    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b8ab25_5d40_45fc_8a55_9ce13bcf8646.slice/crio-05251e6507eb01a66567c2660d64a03d4e5708f45015c5801fb0f319dda4e8fa WatchSource:0}: Error finding container 05251e6507eb01a66567c2660d64a03d4e5708f45015c5801fb0f319dda4e8fa: Status 404 returned error can't find the container with id 05251e6507eb01a66567c2660d64a03d4e5708f45015c5801fb0f319dda4e8fa
Oct 29 23:05:20.996332 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:20.996301    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:21.329269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:21.329191    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2" event={"ID":"44b8ab25-5d40-45fc-8a55-9ce13bcf8646","Type":"ContainerStarted","Data":"c896946be39f6d394910c09be61848d09757e19d3475271ae3912a6af5b510c7"}
Oct 29 23:05:21.329269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:21.329249    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2" event={"ID":"44b8ab25-5d40-45fc-8a55-9ce13bcf8646","Type":"ContainerStarted","Data":"05251e6507eb01a66567c2660d64a03d4e5708f45015c5801fb0f319dda4e8fa"}
Oct 29 23:05:21.329697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:21.329272    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2"
Oct 29 23:05:21.370853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:21.370787    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2" podStartSLOduration=1.370772305 podStartE2EDuration="1.370772305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:21.369796572 +0000 UTC m=+1324.712745319" watchObservedRunningTime="2026-10-29 23:05:21.370772305 +0000 UTC m=+1324.713721074"
Oct 29 23:05:21.526796 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:21.526766    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:22.593017 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:22.592980    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:22.705563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:22.705523    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:22.871870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:22.861029    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd"]
Oct 29 23:05:22.954458 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:22.954420    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:22.973966 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:22.966480    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd"]
Oct 29 23:05:22.973966 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:22.966613    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd"
Oct 29 23:05:22.982295 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:22.982261    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-dfedfe-dockercfg-tphbm\""
Oct 29 23:05:22.982443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:22.982368    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-dfedfe\""
Oct 29 23:05:23.107560 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:23.107523    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kxjw\" (UniqueName: \"kubernetes.io/projected/76880e88-0196-4501-b499-52a0800e26dc-kube-api-access-5kxjw\") pod \"payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd\" (UID: \"76880e88-0196-4501-b499-52a0800e26dc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd"
Oct 29 23:05:23.107771 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:23.107588    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76880e88-0196-4501-b499-52a0800e26dc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd\" (UID: \"76880e88-0196-4501-b499-52a0800e26dc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd"
Oct 29 23:05:23.209762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:23.209673    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kxjw\" (UniqueName: \"kubernetes.io/projected/76880e88-0196-4501-b499-52a0800e26dc-kube-api-access-5kxjw\") pod \"payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd\" (UID: \"76880e88-0196-4501-b499-52a0800e26dc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd"
Oct 29 23:05:23.209762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:23.209729    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76880e88-0196-4501-b499-52a0800e26dc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd\" (UID: \"76880e88-0196-4501-b499-52a0800e26dc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd"
Oct 29 23:05:23.210391 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:23.210372    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76880e88-0196-4501-b499-52a0800e26dc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd\" (UID: \"76880e88-0196-4501-b499-52a0800e26dc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd"
Oct 29 23:05:23.232937 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:23.232904    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kxjw\" (UniqueName: \"kubernetes.io/projected/76880e88-0196-4501-b499-52a0800e26dc-kube-api-access-5kxjw\") pod \"payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd\" (UID: \"76880e88-0196-4501-b499-52a0800e26dc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd"
Oct 29 23:05:23.248018 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:23.247979    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56"
Oct 29 23:05:23.257905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:23.257877    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg"
Oct 29 23:05:23.294749 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:23.294718    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd"
Oct 29 23:05:23.325250 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:23.325220    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:23.642709 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:23.642680    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd"]
Oct 29 23:05:23.644590 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:05:23.644563    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76880e88_0196_4501_b499_52a0800e26dc.slice/crio-dc39bfa107aecae7e0f25cce87bc3102f83782575a4e4a01bcebe9733091b271 WatchSource:0}: Error finding container dc39bfa107aecae7e0f25cce87bc3102f83782575a4e4a01bcebe9733091b271: Status 404 returned error can't find the container with id dc39bfa107aecae7e0f25cce87bc3102f83782575a4e4a01bcebe9733091b271
Oct 29 23:05:23.855638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:23.855608    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:24.004848 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:24.004732    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:24.378896 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:24.378860    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd" event={"ID":"76880e88-0196-4501-b499-52a0800e26dc","Type":"ContainerStarted","Data":"b50b0d3f7b1223fe2bef44051f30b5116877359b5a40c9261fcc8afe7f856dd4"}
Oct 29 23:05:24.379068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:24.378917    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd" event={"ID":"76880e88-0196-4501-b499-52a0800e26dc","Type":"ContainerStarted","Data":"dc39bfa107aecae7e0f25cce87bc3102f83782575a4e4a01bcebe9733091b271"}
Oct 29 23:05:24.379068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:24.378945    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd"
Oct 29 23:05:24.422202 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:24.422151    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd" podStartSLOduration=2.422137558 podStartE2EDuration="2.422137558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:24.4189086 +0000 UTC m=+1327.761857365" watchObservedRunningTime="2026-10-29 23:05:24.422137558 +0000 UTC m=+1327.765086298"
Oct 29 23:05:24.846442 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:24.846361    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"]
Oct 29 23:05:24.851393 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:24.851370    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:05:24.865995 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:24.865971    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-0941c8-serving-cert\""
Oct 29 23:05:24.871525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:24.871502    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"]
Oct 29 23:05:24.950819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:24.950774    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb5221e1-3731-4150-aaba-89507c4b0cc8-metrics-certs\") pod \"maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls\" (UID: \"cb5221e1-3731-4150-aaba-89507c4b0cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:05:24.950986 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:24.950881    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5221e1-3731-4150-aaba-89507c4b0cc8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls\" (UID: \"cb5221e1-3731-4150-aaba-89507c4b0cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:05:24.950986 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:24.950951    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb5221e1-3731-4150-aaba-89507c4b0cc8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls\" (UID: \"cb5221e1-3731-4150-aaba-89507c4b0cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:05:24.951134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:24.951119    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdzzh\" (UniqueName: \"kubernetes.io/projected/cb5221e1-3731-4150-aaba-89507c4b0cc8-kube-api-access-bdzzh\") pod \"maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls\" (UID: \"cb5221e1-3731-4150-aaba-89507c4b0cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:05:25.014491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:25.014443    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:25.052240 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:25.052202    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb5221e1-3731-4150-aaba-89507c4b0cc8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls\" (UID: \"cb5221e1-3731-4150-aaba-89507c4b0cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:05:25.052388 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:05:25.052369    5428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-serving-cert: secret "maas-api-e2e-praxis-a-0941c8-serving-cert" not found
Oct 29 23:05:25.052455 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:25.052415    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdzzh\" (UniqueName: \"kubernetes.io/projected/cb5221e1-3731-4150-aaba-89507c4b0cc8-kube-api-access-bdzzh\") pod \"maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls\" (UID: \"cb5221e1-3731-4150-aaba-89507c4b0cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:05:25.052455 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:05:25.052429    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb5221e1-3731-4150-aaba-89507c4b0cc8-maas-api-tls podName:cb5221e1-3731-4150-aaba-89507c4b0cc8 nodeName:}" failed. No retries permitted until 2026-10-29 23:05:25.552413715 +0000 UTC m=+1328.895362464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb5221e1-3731-4150-aaba-89507c4b0cc8-maas-api-tls") pod "maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls" (UID: "cb5221e1-3731-4150-aaba-89507c4b0cc8") : secret "maas-api-e2e-praxis-a-0941c8-serving-cert" not found
Oct 29 23:05:25.052659 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:25.052643    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb5221e1-3731-4150-aaba-89507c4b0cc8-metrics-certs\") pod \"maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls\" (UID: \"cb5221e1-3731-4150-aaba-89507c4b0cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:05:25.052722 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:25.052686    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5221e1-3731-4150-aaba-89507c4b0cc8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls\" (UID: \"cb5221e1-3731-4150-aaba-89507c4b0cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:05:25.053351 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:25.053328    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5221e1-3731-4150-aaba-89507c4b0cc8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls\" (UID: \"cb5221e1-3731-4150-aaba-89507c4b0cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:05:25.055687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:25.055522    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb5221e1-3731-4150-aaba-89507c4b0cc8-metrics-certs\") pod \"maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls\" (UID: \"cb5221e1-3731-4150-aaba-89507c4b0cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:05:25.075397 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:25.075369    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdzzh\" (UniqueName: \"kubernetes.io/projected/cb5221e1-3731-4150-aaba-89507c4b0cc8-kube-api-access-bdzzh\") pod \"maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls\" (UID: \"cb5221e1-3731-4150-aaba-89507c4b0cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:05:25.129891 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:25.129607    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:25.338540 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:25.338509    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:25.561258 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:25.561183    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb5221e1-3731-4150-aaba-89507c4b0cc8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls\" (UID: \"cb5221e1-3731-4150-aaba-89507c4b0cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:05:25.563853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:25.563822    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb5221e1-3731-4150-aaba-89507c4b0cc8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls\" (UID: \"cb5221e1-3731-4150-aaba-89507c4b0cc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:05:25.765580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:25.765542    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:05:26.091598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:26.091566    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"]
Oct 29 23:05:26.092247 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:05:26.092212    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5221e1_3731_4150_aaba_89507c4b0cc8.slice/crio-1f59d49fbe90ba6e3bb69f84cc42972b5b81699b836c3ce13da1db64c2aa8a55 WatchSource:0}: Error finding container 1f59d49fbe90ba6e3bb69f84cc42972b5b81699b836c3ce13da1db64c2aa8a55: Status 404 returned error can't find the container with id 1f59d49fbe90ba6e3bb69f84cc42972b5b81699b836c3ce13da1db64c2aa8a55
Oct 29 23:05:26.093762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:26.093741    5428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:05:26.262569 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:26.262508    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:26.408828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:26.408780    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls" event={"ID":"cb5221e1-3731-4150-aaba-89507c4b0cc8","Type":"ContainerStarted","Data":"1f59d49fbe90ba6e3bb69f84cc42972b5b81699b836c3ce13da1db64c2aa8a55"}
Oct 29 23:05:27.197515 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:27.197478    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:27.417566 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:27.417529    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls" event={"ID":"cb5221e1-3731-4150-aaba-89507c4b0cc8","Type":"ContainerStarted","Data":"67fe58e8ecb908ff49c81e554a6e7d7f667239ab7edcea1abda8834cfd4a82a8"}
Oct 29 23:05:27.417778 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:27.417617    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:05:27.462184 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:27.462078    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls" podStartSLOduration=3.0491400889999998 podStartE2EDuration="3.462062138s" totalImagesPullingTime="412.922049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:27.458259732 +0000 UTC m=+1330.801208477" watchObservedRunningTime="2026-10-29 23:05:27.462062138 +0000 UTC m=+1330.805010883"
Oct 29 23:05:29.556445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:29.556403    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:30.876441 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:30.876374    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz"]
Oct 29 23:05:30.908845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:30.908813    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w"]
Oct 29 23:05:31.057888 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:31.057830    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:31.218501 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:31.218412    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:31.954438 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:31.954401    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:32.340730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:32.340649    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2"
Oct 29 23:05:33.429847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:33.429815    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:05:33.746235 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:33.746147    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"]
Oct 29 23:05:33.750546 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:33.750528    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"
Oct 29 23:05:33.765743 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:33.765717    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-618d6c-serving-cert\""
Oct 29 23:05:33.780404 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:33.780379    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"]
Oct 29 23:05:33.896870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:33.896836    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:33.900390 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:33.900360    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pdl4\" (UniqueName: \"kubernetes.io/projected/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-kube-api-access-4pdl4\") pod \"maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss\" (UID: \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"
Oct 29 23:05:33.900522 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:33.900466    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-metrics-certs\") pod \"maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss\" (UID: \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"
Oct 29 23:05:33.900522 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:33.900498    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss\" (UID: \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"
Oct 29 23:05:33.900701 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:33.900680    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss\" (UID: \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"
Oct 29 23:05:34.003050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:34.002951    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss\" (UID: \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"
Oct 29 23:05:34.003263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:34.003216    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdl4\" (UniqueName: \"kubernetes.io/projected/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-kube-api-access-4pdl4\") pod \"maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss\" (UID: \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"
Oct 29 23:05:34.003434 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:34.003415    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-metrics-certs\") pod \"maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss\" (UID: \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"
Oct 29 23:05:34.003528 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:34.003478    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss\" (UID: \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"
Oct 29 23:05:34.003703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:34.003677    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss\" (UID: \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"
Oct 29 23:05:34.006572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:34.006533    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss\" (UID: \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"
Oct 29 23:05:34.007594 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:34.007564    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-metrics-certs\") pod \"maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss\" (UID: \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"
Oct 29 23:05:34.011777 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:34.011747    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:34.027140 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:34.027115    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pdl4\" (UniqueName: \"kubernetes.io/projected/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-kube-api-access-4pdl4\") pod \"maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss\" (UID: \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"
Oct 29 23:05:34.064509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:34.064479    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"
Oct 29 23:05:34.427418 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:05:34.427378    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f431b2f_78fd_43ea_bcbf_3d6a4a894f1d.slice/crio-375334d96e06042610406830fb5f0f52a4705e9100bd9f0be6ba51c215e5939f WatchSource:0}: Error finding container 375334d96e06042610406830fb5f0f52a4705e9100bd9f0be6ba51c215e5939f: Status 404 returned error can't find the container with id 375334d96e06042610406830fb5f0f52a4705e9100bd9f0be6ba51c215e5939f
Oct 29 23:05:34.428886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:34.428781    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"]
Oct 29 23:05:34.496065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:34.496020    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss" event={"ID":"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d","Type":"ContainerStarted","Data":"375334d96e06042610406830fb5f0f52a4705e9100bd9f0be6ba51c215e5939f"}
Oct 29 23:05:35.393680 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:35.393647    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd"
Oct 29 23:05:35.512787 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:35.512654    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss" event={"ID":"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d","Type":"ContainerStarted","Data":"2816a900341e1a9b2e673418e938bacdcfce49bfaa905fb9ae2cafe851a76964"}
Oct 29 23:05:35.513395 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:35.512872    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"
Oct 29 23:05:35.561434 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:35.561351    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss" podStartSLOduration=2.229225268 podStartE2EDuration="2.561330133s" totalImagesPullingTime="332.104865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:35.554238366 +0000 UTC m=+1338.897187112" watchObservedRunningTime="2026-10-29 23:05:35.561330133 +0000 UTC m=+1338.904278879"
Oct 29 23:05:35.876682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:35.876647    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz" podUID="aca44441-e031-4a00-8814-6b5e193b0f4a" containerName="payload-processing" containerID="cri-o://ebede9e5286498fc132dcf59e66f0e5235f3d06db0c5f6ad99ac128838a494ec" gracePeriod=25
Oct 29 23:05:35.909324 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:35.909286    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w" podUID="7b8e1b51-65ba-47f7-b7f5-df5fb69a0203" containerName="payload-pre-processing" containerID="cri-o://a49df011d0b166a6844b6893ad41cec7cb85a8362d63d6b5c4782a6137b87198" gracePeriod=25
Oct 29 23:05:36.231181 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.231151    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz"
Oct 29 23:05:36.259963 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.259936    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca44441-e031-4a00-8814-6b5e193b0f4a-kube-api-access-j48cc\" (UniqueName: \"kubernetes.io/projected/aca44441-e031-4a00-8814-6b5e193b0f4a-kube-api-access-j48cc\") pod \"aca44441-e031-4a00-8814-6b5e193b0f4a\" (UID: \"aca44441-e031-4a00-8814-6b5e193b0f4a\") "
Oct 29 23:05:36.260142 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.260128    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aca44441-e031-4a00-8814-6b5e193b0f4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca44441-e031-4a00-8814-6b5e193b0f4a-plugins-config-volume\") pod \"aca44441-e031-4a00-8814-6b5e193b0f4a\" (UID: \"aca44441-e031-4a00-8814-6b5e193b0f4a\") "
Oct 29 23:05:36.260491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.260456    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca44441-e031-4a00-8814-6b5e193b0f4a-plugins-config-volume" pod "aca44441-e031-4a00-8814-6b5e193b0f4a" (UID: "aca44441-e031-4a00-8814-6b5e193b0f4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:36.260728 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.260693    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca44441-e031-4a00-8814-6b5e193b0f4a-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.262165 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.262141    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca44441-e031-4a00-8814-6b5e193b0f4a-kube-api-access-j48cc" pod "aca44441-e031-4a00-8814-6b5e193b0f4a" (UID: "aca44441-e031-4a00-8814-6b5e193b0f4a"). InnerVolumeSpecName "kube-api-access-j48cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:36.262786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.262770    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w"
Oct 29 23:05:36.362207 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.362174    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b8e1b51-65ba-47f7-b7f5-df5fb69a0203-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8e1b51-65ba-47f7-b7f5-df5fb69a0203-plugins-config-volume\") pod \"7b8e1b51-65ba-47f7-b7f5-df5fb69a0203\" (UID: \"7b8e1b51-65ba-47f7-b7f5-df5fb69a0203\") "
Oct 29 23:05:36.362374 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.362351    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8e1b51-65ba-47f7-b7f5-df5fb69a0203-kube-api-access-vkmc5\" (UniqueName: \"kubernetes.io/projected/7b8e1b51-65ba-47f7-b7f5-df5fb69a0203-kube-api-access-vkmc5\") pod \"7b8e1b51-65ba-47f7-b7f5-df5fb69a0203\" (UID: \"7b8e1b51-65ba-47f7-b7f5-df5fb69a0203\") "
Oct 29 23:05:36.362564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.362539    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8e1b51-65ba-47f7-b7f5-df5fb69a0203-plugins-config-volume" pod "7b8e1b51-65ba-47f7-b7f5-df5fb69a0203" (UID: "7b8e1b51-65ba-47f7-b7f5-df5fb69a0203"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:36.362879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.362861    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j48cc\" (UniqueName: \"kubernetes.io/projected/aca44441-e031-4a00-8814-6b5e193b0f4a-kube-api-access-j48cc\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.362936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.362882    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8e1b51-65ba-47f7-b7f5-df5fb69a0203-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.364430 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.364409    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8e1b51-65ba-47f7-b7f5-df5fb69a0203-kube-api-access-vkmc5" pod "7b8e1b51-65ba-47f7-b7f5-df5fb69a0203" (UID: "7b8e1b51-65ba-47f7-b7f5-df5fb69a0203"). InnerVolumeSpecName "kube-api-access-vkmc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:36.445978 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.445900    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:36.464484 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.464452    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkmc5\" (UniqueName: \"kubernetes.io/projected/7b8e1b51-65ba-47f7-b7f5-df5fb69a0203-kube-api-access-vkmc5\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.523035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.522992    5428 generic.go:356] "Generic (PLEG): container finished" podID="7b8e1b51-65ba-47f7-b7f5-df5fb69a0203" containerID="a49df011d0b166a6844b6893ad41cec7cb85a8362d63d6b5c4782a6137b87198" exitCode=0
Oct 29 23:05:36.523035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.523037    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w"
Oct 29 23:05:36.523530 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.523032    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w" event={"ID":"7b8e1b51-65ba-47f7-b7f5-df5fb69a0203","Type":"ContainerDied","Data":"a49df011d0b166a6844b6893ad41cec7cb85a8362d63d6b5c4782a6137b87198"}
Oct 29 23:05:36.523530 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.523196    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w" event={"ID":"7b8e1b51-65ba-47f7-b7f5-df5fb69a0203","Type":"ContainerDied","Data":"83b8af6bac47b40ebf60b0e79824967c43796f09d903dcc239c94965706653b6"}
Oct 29 23:05:36.523530 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.523233    5428 scope.go:122] "RemoveContainer" containerID="a49df011d0b166a6844b6893ad41cec7cb85a8362d63d6b5c4782a6137b87198"
Oct 29 23:05:36.525637 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.525609    5428 generic.go:356] "Generic (PLEG): container finished" podID="aca44441-e031-4a00-8814-6b5e193b0f4a" containerID="ebede9e5286498fc132dcf59e66f0e5235f3d06db0c5f6ad99ac128838a494ec" exitCode=0
Oct 29 23:05:36.525776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.525650    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz"
Oct 29 23:05:36.525776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.525698    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz" event={"ID":"aca44441-e031-4a00-8814-6b5e193b0f4a","Type":"ContainerDied","Data":"ebede9e5286498fc132dcf59e66f0e5235f3d06db0c5f6ad99ac128838a494ec"}
Oct 29 23:05:36.525776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.525755    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz" event={"ID":"aca44441-e031-4a00-8814-6b5e193b0f4a","Type":"ContainerDied","Data":"08fd6fdb490b727d8f7f3e22d8ee06682673635bde57c9e2426e77b51d8b299a"}
Oct 29 23:05:36.539714 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.539695    5428 scope.go:122] "RemoveContainer" containerID="a49df011d0b166a6844b6893ad41cec7cb85a8362d63d6b5c4782a6137b87198"
Oct 29 23:05:36.540108 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:05:36.540085    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49df011d0b166a6844b6893ad41cec7cb85a8362d63d6b5c4782a6137b87198\": container with ID starting with a49df011d0b166a6844b6893ad41cec7cb85a8362d63d6b5c4782a6137b87198 not found: ID does not exist" containerID="a49df011d0b166a6844b6893ad41cec7cb85a8362d63d6b5c4782a6137b87198"
Oct 29 23:05:36.540217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.540123    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49df011d0b166a6844b6893ad41cec7cb85a8362d63d6b5c4782a6137b87198"} err="failed to get container status \"a49df011d0b166a6844b6893ad41cec7cb85a8362d63d6b5c4782a6137b87198\": rpc error: code = NotFound desc = could not find container \"a49df011d0b166a6844b6893ad41cec7cb85a8362d63d6b5c4782a6137b87198\": container with ID starting with a49df011d0b166a6844b6893ad41cec7cb85a8362d63d6b5c4782a6137b87198 not found: ID does not exist"
Oct 29 23:05:36.540217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.540152    5428 scope.go:122] "RemoveContainer" containerID="ebede9e5286498fc132dcf59e66f0e5235f3d06db0c5f6ad99ac128838a494ec"
Oct 29 23:05:36.553219 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.553155    5428 scope.go:122] "RemoveContainer" containerID="ebede9e5286498fc132dcf59e66f0e5235f3d06db0c5f6ad99ac128838a494ec"
Oct 29 23:05:36.554256 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:05:36.554219    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebede9e5286498fc132dcf59e66f0e5235f3d06db0c5f6ad99ac128838a494ec\": container with ID starting with ebede9e5286498fc132dcf59e66f0e5235f3d06db0c5f6ad99ac128838a494ec not found: ID does not exist" containerID="ebede9e5286498fc132dcf59e66f0e5235f3d06db0c5f6ad99ac128838a494ec"
Oct 29 23:05:36.554390 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.554280    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebede9e5286498fc132dcf59e66f0e5235f3d06db0c5f6ad99ac128838a494ec"} err="failed to get container status \"ebede9e5286498fc132dcf59e66f0e5235f3d06db0c5f6ad99ac128838a494ec\": rpc error: code = NotFound desc = could not find container \"ebede9e5286498fc132dcf59e66f0e5235f3d06db0c5f6ad99ac128838a494ec\": container with ID starting with ebede9e5286498fc132dcf59e66f0e5235f3d06db0c5f6ad99ac128838a494ec not found: ID does not exist"
Oct 29 23:05:36.594904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.594871    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w"]
Oct 29 23:05:36.600603 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.600544    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-41b12c-d6f9ccbc5-t654w"]
Oct 29 23:05:36.646562 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.646525    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz"]
Oct 29 23:05:36.652852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.652821    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-41b12c-5f5499869b-l9dgz"]
Oct 29 23:05:36.779053 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.778981    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b8e1b51-65ba-47f7-b7f5-df5fb69a0203" path="/var/lib/kubelet/pods/7b8e1b51-65ba-47f7-b7f5-df5fb69a0203/volumes"
Oct 29 23:05:36.779460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:36.779445    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca44441-e031-4a00-8814-6b5e193b0f4a" path="/var/lib/kubelet/pods/aca44441-e031-4a00-8814-6b5e193b0f4a/volumes"
Oct 29 23:05:38.494895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:38.494856    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:41.060859 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:41.060781    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:41.532014 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:41.531984    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"
Oct 29 23:05:42.953955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:42.953921    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:44.466279 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.466238    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"]
Oct 29 23:05:44.468241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.468210    5428 cpu_manager.go:414] "containerMap: removing container" podUID="7b8e1b51-65ba-47f7-b7f5-df5fb69a0203" containerName="payload-pre-processing"
Oct 29 23:05:44.468241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.468238    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b8e1b51-65ba-47f7-b7f5-df5fb69a0203" containerName="payload-pre-processing"
Oct 29 23:05:44.469130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.468270    5428 cpu_manager.go:414] "containerMap: removing container" podUID="aca44441-e031-4a00-8814-6b5e193b0f4a" containerName="payload-processing"
Oct 29 23:05:44.469130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.468277    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca44441-e031-4a00-8814-6b5e193b0f4a" containerName="payload-processing"
Oct 29 23:05:44.469130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.468666    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b8e1b51-65ba-47f7-b7f5-df5fb69a0203" containerName="payload-pre-processing"
Oct 29 23:05:44.469130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.468691    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca44441-e031-4a00-8814-6b5e193b0f4a" containerName="payload-processing"
Oct 29 23:05:44.475488 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.475459    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"
Oct 29 23:05:44.493684 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.493657    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-57b28e-serving-cert\""
Oct 29 23:05:44.494020 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.493994    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"]
Oct 29 23:05:44.555520 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.555485    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:44.590948 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.590915    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-metrics-certs\") pod \"maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8\" (UID: \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"
Oct 29 23:05:44.591149 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.591129    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-openshift-service-ca\") pod \"maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8\" (UID: \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"
Oct 29 23:05:44.591221 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.591168    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng6sm\" (UniqueName: \"kubernetes.io/projected/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-kube-api-access-ng6sm\") pod \"maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8\" (UID: \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"
Oct 29 23:05:44.591291 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.591276    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-maas-api-tls\") pod \"maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8\" (UID: \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"
Oct 29 23:05:44.671142 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.671110    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:44.692860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.692783    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-openshift-service-ca\") pod \"maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8\" (UID: \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"
Oct 29 23:05:44.693233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.693095    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng6sm\" (UniqueName: \"kubernetes.io/projected/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-kube-api-access-ng6sm\") pod \"maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8\" (UID: \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"
Oct 29 23:05:44.693759 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.693732    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-maas-api-tls\") pod \"maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8\" (UID: \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"
Oct 29 23:05:44.693940 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.693908    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-metrics-certs\") pod \"maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8\" (UID: \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"
Oct 29 23:05:44.694205 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.694140    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-openshift-service-ca\") pod \"maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8\" (UID: \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"
Oct 29 23:05:44.703074 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.703043    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-maas-api-tls\") pod \"maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8\" (UID: \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"
Oct 29 23:05:44.713390 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.713327    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-metrics-certs\") pod \"maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8\" (UID: \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"
Oct 29 23:05:44.725493 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.725413    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng6sm\" (UniqueName: \"kubernetes.io/projected/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-kube-api-access-ng6sm\") pod \"maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8\" (UID: \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"
Oct 29 23:05:44.792127 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.792095    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:44.794066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:44.794039    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"
Oct 29 23:05:45.116618 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:45.116589    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"]
Oct 29 23:05:45.117285 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:05:45.117256    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f472b3_56b6_4901_a1a9_c4d0c07c6ad7.slice/crio-de86ac9e5da079f3a93079bbf7076d8af470ac6f80409f4159785f9c5ee16c59 WatchSource:0}: Error finding container de86ac9e5da079f3a93079bbf7076d8af470ac6f80409f4159785f9c5ee16c59: Status 404 returned error can't find the container with id de86ac9e5da079f3a93079bbf7076d8af470ac6f80409f4159785f9c5ee16c59
Oct 29 23:05:45.600524 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:45.600492    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8" event={"ID":"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7","Type":"ContainerStarted","Data":"de86ac9e5da079f3a93079bbf7076d8af470ac6f80409f4159785f9c5ee16c59"}
Oct 29 23:05:46.486296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:46.486261    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:46.611058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:46.611015    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8" event={"ID":"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7","Type":"ContainerStarted","Data":"fc40ff8138d775f46cd7579a7c4e94c89615a314be9cfb08dbca812e29e9794a"}
Oct 29 23:05:46.611654 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:46.611338    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"
Oct 29 23:05:46.656235 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:46.656163    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8" podStartSLOduration=2.215886885 podStartE2EDuration="2.656141858s" totalImagesPullingTime="440.254973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:46.654856233 +0000 UTC m=+1349.997805005" watchObservedRunningTime="2026-10-29 23:05:46.656141858 +0000 UTC m=+1349.999090606"
Oct 29 23:05:47.368842 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:05:47.368774    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:05:48.227530 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:48.227500    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:49.736885 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:49.736844    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"]
Oct 29 23:05:49.737361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:49.737140    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl" podUID="d629caba-fbff-4186-9a17-e2ad12edb908" containerName="maas-api" containerID="cri-o://b0f8bb9334deb0078b7398e92abc67336ec4b50c98c43debd414fd65340e7ab8" gracePeriod=30
Oct 29 23:05:49.810560 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:49.810529    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:50.092354 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.092324    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:05:50.292992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.281596    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d629caba-fbff-4186-9a17-e2ad12edb908-kube-api-access-l7nj5\" (UniqueName: \"kubernetes.io/projected/d629caba-fbff-4186-9a17-e2ad12edb908-kube-api-access-l7nj5\") pod \"d629caba-fbff-4186-9a17-e2ad12edb908\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") "
Oct 29 23:05:50.292992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.281725    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d629caba-fbff-4186-9a17-e2ad12edb908-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d629caba-fbff-4186-9a17-e2ad12edb908-openshift-service-ca\") pod \"d629caba-fbff-4186-9a17-e2ad12edb908\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") "
Oct 29 23:05:50.292992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.281768    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-metrics-certs\") pod \"d629caba-fbff-4186-9a17-e2ad12edb908\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") "
Oct 29 23:05:50.292992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.281959    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-maas-api-tls\") pod \"d629caba-fbff-4186-9a17-e2ad12edb908\" (UID: \"d629caba-fbff-4186-9a17-e2ad12edb908\") "
Oct 29 23:05:50.292992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.282174    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d629caba-fbff-4186-9a17-e2ad12edb908-openshift-service-ca" pod "d629caba-fbff-4186-9a17-e2ad12edb908" (UID: "d629caba-fbff-4186-9a17-e2ad12edb908"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:50.292992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.282669    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d629caba-fbff-4186-9a17-e2ad12edb908-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:50.292992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.285113    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-maas-api-tls" pod "d629caba-fbff-4186-9a17-e2ad12edb908" (UID: "d629caba-fbff-4186-9a17-e2ad12edb908"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:50.292992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.286221    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-metrics-certs" pod "d629caba-fbff-4186-9a17-e2ad12edb908" (UID: "d629caba-fbff-4186-9a17-e2ad12edb908"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:50.292992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.286504    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d629caba-fbff-4186-9a17-e2ad12edb908-kube-api-access-l7nj5" pod "d629caba-fbff-4186-9a17-e2ad12edb908" (UID: "d629caba-fbff-4186-9a17-e2ad12edb908"). InnerVolumeSpecName "kube-api-access-l7nj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:50.384354 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.384323    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:50.384499 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.384361    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7nj5\" (UniqueName: \"kubernetes.io/projected/d629caba-fbff-4186-9a17-e2ad12edb908-kube-api-access-l7nj5\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:50.384499 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.384375    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d629caba-fbff-4186-9a17-e2ad12edb908-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:05:50.649879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.649753    5428 generic.go:356] "Generic (PLEG): container finished" podID="d629caba-fbff-4186-9a17-e2ad12edb908" containerID="b0f8bb9334deb0078b7398e92abc67336ec4b50c98c43debd414fd65340e7ab8" exitCode=0
Oct 29 23:05:50.649879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.649843    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"
Oct 29 23:05:50.650109 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.649840    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl" event={"ID":"d629caba-fbff-4186-9a17-e2ad12edb908","Type":"ContainerDied","Data":"b0f8bb9334deb0078b7398e92abc67336ec4b50c98c43debd414fd65340e7ab8"}
Oct 29 23:05:50.650109 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.649974    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl" event={"ID":"d629caba-fbff-4186-9a17-e2ad12edb908","Type":"ContainerDied","Data":"2170b87cd3551963e96870c9c7b5e4bb2b9996fa86abbd275d051ecffcb012cd"}
Oct 29 23:05:50.650109 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.650002    5428 scope.go:122] "RemoveContainer" containerID="b0f8bb9334deb0078b7398e92abc67336ec4b50c98c43debd414fd65340e7ab8"
Oct 29 23:05:50.667091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.666976    5428 scope.go:122] "RemoveContainer" containerID="b0f8bb9334deb0078b7398e92abc67336ec4b50c98c43debd414fd65340e7ab8"
Oct 29 23:05:50.667442 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:05:50.667375    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f8bb9334deb0078b7398e92abc67336ec4b50c98c43debd414fd65340e7ab8\": container with ID starting with b0f8bb9334deb0078b7398e92abc67336ec4b50c98c43debd414fd65340e7ab8 not found: ID does not exist" containerID="b0f8bb9334deb0078b7398e92abc67336ec4b50c98c43debd414fd65340e7ab8"
Oct 29 23:05:50.667442 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.667416    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f8bb9334deb0078b7398e92abc67336ec4b50c98c43debd414fd65340e7ab8"} err="failed to get container status \"b0f8bb9334deb0078b7398e92abc67336ec4b50c98c43debd414fd65340e7ab8\": rpc error: code = NotFound desc = could not find container \"b0f8bb9334deb0078b7398e92abc67336ec4b50c98c43debd414fd65340e7ab8\": container with ID starting with b0f8bb9334deb0078b7398e92abc67336ec4b50c98c43debd414fd65340e7ab8 not found: ID does not exist"
Oct 29 23:05:50.709328 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.709292    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"]
Oct 29 23:05:50.720710 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.720627    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d430df-65fd7555cc-wg8bl"]
Oct 29 23:05:50.779563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.779532    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d629caba-fbff-4186-9a17-e2ad12edb908" path="/var/lib/kubelet/pods/d629caba-fbff-4186-9a17-e2ad12edb908/volumes"
Oct 29 23:05:50.899253 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:50.899211    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:51.040544 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:51.040464    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:51.165651 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:51.165618    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:51.621975 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:51.621941    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:52.625247 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.625213    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"
Oct 29 23:05:52.640473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.640438    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"]
Oct 29 23:05:52.641297 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.641274    5428 cpu_manager.go:414] "containerMap: removing container" podUID="d629caba-fbff-4186-9a17-e2ad12edb908" containerName="maas-api"
Oct 29 23:05:52.641297 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.641295    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d629caba-fbff-4186-9a17-e2ad12edb908" containerName="maas-api"
Oct 29 23:05:52.641477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.641466    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d629caba-fbff-4186-9a17-e2ad12edb908" containerName="maas-api"
Oct 29 23:05:52.646791 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.646771    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:05:52.662750 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.662724    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-ce5978-serving-cert\""
Oct 29 23:05:52.667359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.667334    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"]
Oct 29 23:05:52.716841 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.716769    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4pd9\" (UniqueName: \"kubernetes.io/projected/3bca0741-fd7f-4183-b9f7-f5c827532f72-kube-api-access-l4pd9\") pod \"maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j\" (UID: \"3bca0741-fd7f-4183-b9f7-f5c827532f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:05:52.717029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.717008    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bca0741-fd7f-4183-b9f7-f5c827532f72-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j\" (UID: \"3bca0741-fd7f-4183-b9f7-f5c827532f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:05:52.717193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.717165    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bca0741-fd7f-4183-b9f7-f5c827532f72-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j\" (UID: \"3bca0741-fd7f-4183-b9f7-f5c827532f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:05:52.717335 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.717239    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bca0741-fd7f-4183-b9f7-f5c827532f72-metrics-certs\") pod \"maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j\" (UID: \"3bca0741-fd7f-4183-b9f7-f5c827532f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:05:52.818831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.818756    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bca0741-fd7f-4183-b9f7-f5c827532f72-metrics-certs\") pod \"maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j\" (UID: \"3bca0741-fd7f-4183-b9f7-f5c827532f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:05:52.819086 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.819063    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4pd9\" (UniqueName: \"kubernetes.io/projected/3bca0741-fd7f-4183-b9f7-f5c827532f72-kube-api-access-l4pd9\") pod \"maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j\" (UID: \"3bca0741-fd7f-4183-b9f7-f5c827532f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:05:52.819236 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.819172    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bca0741-fd7f-4183-b9f7-f5c827532f72-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j\" (UID: \"3bca0741-fd7f-4183-b9f7-f5c827532f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:05:52.819313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.819284    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bca0741-fd7f-4183-b9f7-f5c827532f72-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j\" (UID: \"3bca0741-fd7f-4183-b9f7-f5c827532f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:05:52.819528 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:05:52.819477    5428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-serving-cert: secret "maas-api-e2e-praxis-a-ce5978-serving-cert" not found
Oct 29 23:05:52.819640 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:05:52.819546    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bca0741-fd7f-4183-b9f7-f5c827532f72-maas-api-tls podName:3bca0741-fd7f-4183-b9f7-f5c827532f72 nodeName:}" failed. No retries permitted until 2026-10-29 23:05:53.319526218 +0000 UTC m=+1356.662474955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3bca0741-fd7f-4183-b9f7-f5c827532f72-maas-api-tls") pod "maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j" (UID: "3bca0741-fd7f-4183-b9f7-f5c827532f72") : secret "maas-api-e2e-praxis-a-ce5978-serving-cert" not found
Oct 29 23:05:52.820122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.820098    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bca0741-fd7f-4183-b9f7-f5c827532f72-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j\" (UID: \"3bca0741-fd7f-4183-b9f7-f5c827532f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:05:52.821694 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.821659    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bca0741-fd7f-4183-b9f7-f5c827532f72-metrics-certs\") pod \"maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j\" (UID: \"3bca0741-fd7f-4183-b9f7-f5c827532f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:05:52.854389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.854364    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4pd9\" (UniqueName: \"kubernetes.io/projected/3bca0741-fd7f-4183-b9f7-f5c827532f72-kube-api-access-l4pd9\") pod \"maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j\" (UID: \"3bca0741-fd7f-4183-b9f7-f5c827532f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:05:52.900467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:52.900395    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:53.018263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:53.018200    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:53.170779 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:53.170691    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:53.331995 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:53.331964    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bca0741-fd7f-4183-b9f7-f5c827532f72-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j\" (UID: \"3bca0741-fd7f-4183-b9f7-f5c827532f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:05:53.334380 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:53.334353    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bca0741-fd7f-4183-b9f7-f5c827532f72-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j\" (UID: \"3bca0741-fd7f-4183-b9f7-f5c827532f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:05:53.562603 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:53.562512    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:05:53.877054 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:53.877030    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"]
Oct 29 23:05:53.877642 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:05:53.877617    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bca0741_fd7f_4183_b9f7_f5c827532f72.slice/crio-ebd0b6f18b9faa43d22d2a689fff56d7bdb227bff26061ba0da51945e69eaaa9 WatchSource:0}: Error finding container ebd0b6f18b9faa43d22d2a689fff56d7bdb227bff26061ba0da51945e69eaaa9: Status 404 returned error can't find the container with id ebd0b6f18b9faa43d22d2a689fff56d7bdb227bff26061ba0da51945e69eaaa9
Oct 29 23:05:54.694290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:54.694252    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j" event={"ID":"3bca0741-fd7f-4183-b9f7-f5c827532f72","Type":"ContainerStarted","Data":"17529fab1dbfb8332ddc381b82330d3fed385feb6ac37c8c5173146ca3af4242"}
Oct 29 23:05:54.694474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:54.694318    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j" event={"ID":"3bca0741-fd7f-4183-b9f7-f5c827532f72","Type":"ContainerStarted","Data":"ebd0b6f18b9faa43d22d2a689fff56d7bdb227bff26061ba0da51945e69eaaa9"}
Oct 29 23:05:54.694474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:54.694344    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:05:54.737491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:54.737418    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j" podStartSLOduration=2.390686717 podStartE2EDuration="2.737399549s" totalImagesPullingTime="346.712832ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:54.736202629 +0000 UTC m=+1358.079151376" watchObservedRunningTime="2026-10-29 23:05:54.737399549 +0000 UTC m=+1358.080348296"
Oct 29 23:05:54.795757 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:54.795726    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:56.537041 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:56.537004    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:56.742836 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:56.742764    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:56.861042 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:56.860967    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:58.169553 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:58.169474    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:58.302722 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:58.302684    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:58.537300 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:58.537192    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:58.690232 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:58.690190    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:59.507074 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:05:59.507043    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:00.005252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:00.005220    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:00.703587 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:00.703554    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:00.707935 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:00.707914    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:06:01.164238 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:01.164192    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:03.496326 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:03.496292    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:04.009443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:04.009400    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:04.509663 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:04.509622    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:04.643069 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:04.642995    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:04.948362 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:04.948325    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:05.156519 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:05.156478    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:05.272725 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:05.272626    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:05.866703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:05.866669    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:06.588010 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:06.587976    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:06.977594 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:06.977525    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:07.095133 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.095096    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:07.143657 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.143625    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b"]
Oct 29 23:06:07.148265 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.148243    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b"
Oct 29 23:06:07.169245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.169213    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b"]
Oct 29 23:06:07.172726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.172697    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5294cfe-5196-4b3b-bfbe-60afe05d6cb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b\" (UID: \"d5294cfe-5196-4b3b-bfbe-60afe05d6cb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b"
Oct 29 23:06:07.172891 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.172821    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6vr5\" (UniqueName: \"kubernetes.io/projected/d5294cfe-5196-4b3b-bfbe-60afe05d6cb5-kube-api-access-b6vr5\") pod \"payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b\" (UID: \"d5294cfe-5196-4b3b-bfbe-60afe05d6cb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b"
Oct 29 23:06:07.174909 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.174874    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-921680-dockercfg-vd5q5\""
Oct 29 23:06:07.175029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.174910    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-921680\""
Oct 29 23:06:07.242382 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.242306    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:07.246584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.246545    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn"]
Oct 29 23:06:07.252475 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.252447    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn"
Oct 29 23:06:07.275062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.275026    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcks9\" (UniqueName: \"kubernetes.io/projected/847d1425-c094-4abc-b085-55d7c9c2bcbd-kube-api-access-pcks9\") pod \"payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn\" (UID: \"847d1425-c094-4abc-b085-55d7c9c2bcbd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn"
Oct 29 23:06:07.275355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.275331    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/847d1425-c094-4abc-b085-55d7c9c2bcbd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn\" (UID: \"847d1425-c094-4abc-b085-55d7c9c2bcbd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn"
Oct 29 23:06:07.275570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.275550    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5294cfe-5196-4b3b-bfbe-60afe05d6cb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b\" (UID: \"d5294cfe-5196-4b3b-bfbe-60afe05d6cb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b"
Oct 29 23:06:07.275735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.275719    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6vr5\" (UniqueName: \"kubernetes.io/projected/d5294cfe-5196-4b3b-bfbe-60afe05d6cb5-kube-api-access-b6vr5\") pod \"payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b\" (UID: \"d5294cfe-5196-4b3b-bfbe-60afe05d6cb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b"
Oct 29 23:06:07.277269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.277243    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5294cfe-5196-4b3b-bfbe-60afe05d6cb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b\" (UID: \"d5294cfe-5196-4b3b-bfbe-60afe05d6cb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b"
Oct 29 23:06:07.378338 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.363006    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn"]
Oct 29 23:06:07.381159 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.379720    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcks9\" (UniqueName: \"kubernetes.io/projected/847d1425-c094-4abc-b085-55d7c9c2bcbd-kube-api-access-pcks9\") pod \"payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn\" (UID: \"847d1425-c094-4abc-b085-55d7c9c2bcbd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn"
Oct 29 23:06:07.381159 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.379821    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/847d1425-c094-4abc-b085-55d7c9c2bcbd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn\" (UID: \"847d1425-c094-4abc-b085-55d7c9c2bcbd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn"
Oct 29 23:06:07.381159 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.381090    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/847d1425-c094-4abc-b085-55d7c9c2bcbd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn\" (UID: \"847d1425-c094-4abc-b085-55d7c9c2bcbd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn"
Oct 29 23:06:07.384382 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.384330    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6vr5\" (UniqueName: \"kubernetes.io/projected/d5294cfe-5196-4b3b-bfbe-60afe05d6cb5-kube-api-access-b6vr5\") pod \"payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b\" (UID: \"d5294cfe-5196-4b3b-bfbe-60afe05d6cb5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b"
Oct 29 23:06:07.421889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.421856    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcks9\" (UniqueName: \"kubernetes.io/projected/847d1425-c094-4abc-b085-55d7c9c2bcbd-kube-api-access-pcks9\") pod \"payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn\" (UID: \"847d1425-c094-4abc-b085-55d7c9c2bcbd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn"
Oct 29 23:06:07.462994 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.462960    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b"
Oct 29 23:06:07.618203 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.618163    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn"
Oct 29 23:06:07.780315 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.780283    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b"]
Oct 29 23:06:07.781149 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:06:07.781119    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5294cfe_5196_4b3b_bfbe_60afe05d6cb5.slice/crio-62bc3b7e65a54afb0ded0f51f8591de29f1d1bb23ee309c86b4f2c4851ee8b57 WatchSource:0}: Error finding container 62bc3b7e65a54afb0ded0f51f8591de29f1d1bb23ee309c86b4f2c4851ee8b57: Status 404 returned error can't find the container with id 62bc3b7e65a54afb0ded0f51f8591de29f1d1bb23ee309c86b4f2c4851ee8b57
Oct 29 23:06:07.848671 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.848637    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b" event={"ID":"d5294cfe-5196-4b3b-bfbe-60afe05d6cb5","Type":"ContainerStarted","Data":"62bc3b7e65a54afb0ded0f51f8591de29f1d1bb23ee309c86b4f2c4851ee8b57"}
Oct 29 23:06:07.931359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:07.931328    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn"]
Oct 29 23:06:07.932386 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:06:07.932352    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847d1425_c094_4abc_b085_55d7c9c2bcbd.slice/crio-01b633a8954c1a37aa49f453baa515e211eaaada1f607743fcc10be133db42f6 WatchSource:0}: Error finding container 01b633a8954c1a37aa49f453baa515e211eaaada1f607743fcc10be133db42f6: Status 404 returned error can't find the container with id 01b633a8954c1a37aa49f453baa515e211eaaada1f607743fcc10be133db42f6
Oct 29 23:06:08.861870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:08.861699    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b" event={"ID":"d5294cfe-5196-4b3b-bfbe-60afe05d6cb5","Type":"ContainerStarted","Data":"379bcaa13b403ebdbe7f431fc918006e0e4e2ef19ca8cacdbcf45ea3c427c463"}
Oct 29 23:06:08.862556 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:08.862534    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b"
Oct 29 23:06:08.864603 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:08.864561    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn" event={"ID":"847d1425-c094-4abc-b085-55d7c9c2bcbd","Type":"ContainerStarted","Data":"52085e6c6e7a92ecb93e688a6fcffa4a0c43339dd1f9d16f97feb86862c6e936"}
Oct 29 23:06:08.864738 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:08.864621    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn" event={"ID":"847d1425-c094-4abc-b085-55d7c9c2bcbd","Type":"ContainerStarted","Data":"01b633a8954c1a37aa49f453baa515e211eaaada1f607743fcc10be133db42f6"}
Oct 29 23:06:08.864926 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:08.864904    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn"
Oct 29 23:06:08.907290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:08.907238    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b" podStartSLOduration=1.9072254279999998 podStartE2EDuration="1.907225428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:08.903280188 +0000 UTC m=+1372.246228933" watchObservedRunningTime="2026-10-29 23:06:08.907225428 +0000 UTC m=+1372.250174173"
Oct 29 23:06:08.948238 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:08.948193    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn" podStartSLOduration=1.9481810689999999 podStartE2EDuration="1.948181069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:08.944108587 +0000 UTC m=+1372.287057333" watchObservedRunningTime="2026-10-29 23:06:08.948181069 +0000 UTC m=+1372.291129813"
Oct 29 23:06:09.097237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:09.097200    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:10.150747 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:10.150718    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:10.270147 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:10.270110    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:11.199180 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:11.199150    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:11.901258 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:11.901225    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:12.004922 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:12.004891    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2"]
Oct 29 23:06:12.009704 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:12.009681    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2"
Oct 29 23:06:12.031424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:12.031395    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2"]
Oct 29 23:06:12.033504 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:12.033476    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:12.037618 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:12.037590    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-957eef\""
Oct 29 23:06:12.052784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:12.052749    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92ea570b-9973-4fd2-b18a-aa32192ba3d8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2\" (UID: \"92ea570b-9973-4fd2-b18a-aa32192ba3d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2"
Oct 29 23:06:12.053031 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:12.053008    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjr77\" (UniqueName: \"kubernetes.io/projected/92ea570b-9973-4fd2-b18a-aa32192ba3d8-kube-api-access-bjr77\") pod \"payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2\" (UID: \"92ea570b-9973-4fd2-b18a-aa32192ba3d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2"
Oct 29 23:06:12.158898 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:12.158590    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjr77\" (UniqueName: \"kubernetes.io/projected/92ea570b-9973-4fd2-b18a-aa32192ba3d8-kube-api-access-bjr77\") pod \"payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2\" (UID: \"92ea570b-9973-4fd2-b18a-aa32192ba3d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2"
Oct 29 23:06:12.158898 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:12.158786    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92ea570b-9973-4fd2-b18a-aa32192ba3d8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2\" (UID: \"92ea570b-9973-4fd2-b18a-aa32192ba3d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2"
Oct 29 23:06:12.175830 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:12.161264    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92ea570b-9973-4fd2-b18a-aa32192ba3d8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2\" (UID: \"92ea570b-9973-4fd2-b18a-aa32192ba3d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2"
Oct 29 23:06:12.206821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:12.206238    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjr77\" (UniqueName: \"kubernetes.io/projected/92ea570b-9973-4fd2-b18a-aa32192ba3d8-kube-api-access-bjr77\") pod \"payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2\" (UID: \"92ea570b-9973-4fd2-b18a-aa32192ba3d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2"
Oct 29 23:06:12.328250 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:12.328217    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2"
Oct 29 23:06:12.640600 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:12.640569    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2"]
Oct 29 23:06:12.641159 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:06:12.641133    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ea570b_9973_4fd2_b18a_aa32192ba3d8.slice/crio-6d2cdd2e10295139ca2059866c1df3966d6be4b5162bbb06a18daf51b81f8038 WatchSource:0}: Error finding container 6d2cdd2e10295139ca2059866c1df3966d6be4b5162bbb06a18daf51b81f8038: Status 404 returned error can't find the container with id 6d2cdd2e10295139ca2059866c1df3966d6be4b5162bbb06a18daf51b81f8038
Oct 29 23:06:12.899731 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:12.899637    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2" event={"ID":"92ea570b-9973-4fd2-b18a-aa32192ba3d8","Type":"ContainerStarted","Data":"9135a195457e41d388b1a228ab52fbad7c2dfe7f82a52adc1178c20a17c42acf"}
Oct 29 23:06:12.899731 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:12.899700    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2" event={"ID":"92ea570b-9973-4fd2-b18a-aa32192ba3d8","Type":"ContainerStarted","Data":"6d2cdd2e10295139ca2059866c1df3966d6be4b5162bbb06a18daf51b81f8038"}
Oct 29 23:06:12.899980 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:12.899834    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2"
Oct 29 23:06:12.942735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:12.942688    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2" podStartSLOduration=1.942675764 podStartE2EDuration="1.942675764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:12.939268902 +0000 UTC m=+1376.282217647" watchObservedRunningTime="2026-10-29 23:06:12.942675764 +0000 UTC m=+1376.285624524"
Oct 29 23:06:13.570702 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:13.570664    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:14.916910 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:14.916865    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb"]
Oct 29 23:06:14.948311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:14.948274    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j"]
Oct 29 23:06:15.105982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:15.105941    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:15.225895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:15.225782    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:15.376030 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:15.375993    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:16.058419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:16.058367    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:17.377202 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:06:17.377162    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:06:19.394967 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:19.394926    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:19.876006 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:19.875975    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b"
Oct 29 23:06:19.876209 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:19.876081    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn"
Oct 29 23:06:19.917380 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:19.917340    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb" podUID="fbb7c22d-bbfc-4ade-ab22-db401b342897" containerName="payload-processing" containerID="cri-o://d1c59ea676f1602d80b144bf6ff7bcf89327e8707e0206c8050103ff7604e594" gracePeriod=25
Oct 29 23:06:19.948971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:19.948932    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j" podUID="af142501-4ba8-4bb2-a7ce-a1672f90beed" containerName="payload-pre-processing" containerID="cri-o://2314dfb20525de4cc85088333d6e0617f22d546c95404685afa2ec0125632957" gracePeriod=25
Oct 29 23:06:20.262964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.262931    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb"
Oct 29 23:06:20.289164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.289141    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j"
Oct 29 23:06:20.331049 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.331014    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af142501-4ba8-4bb2-a7ce-a1672f90beed-kube-api-access-nn4kf\" (UniqueName: \"kubernetes.io/projected/af142501-4ba8-4bb2-a7ce-a1672f90beed-kube-api-access-nn4kf\") pod \"af142501-4ba8-4bb2-a7ce-a1672f90beed\" (UID: \"af142501-4ba8-4bb2-a7ce-a1672f90beed\") "
Oct 29 23:06:20.331291 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.331270    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb7c22d-bbfc-4ade-ab22-db401b342897-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb7c22d-bbfc-4ade-ab22-db401b342897-plugins-config-volume\") pod \"fbb7c22d-bbfc-4ade-ab22-db401b342897\" (UID: \"fbb7c22d-bbfc-4ade-ab22-db401b342897\") "
Oct 29 23:06:20.331366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.331353    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af142501-4ba8-4bb2-a7ce-a1672f90beed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af142501-4ba8-4bb2-a7ce-a1672f90beed-plugins-config-volume\") pod \"af142501-4ba8-4bb2-a7ce-a1672f90beed\" (UID: \"af142501-4ba8-4bb2-a7ce-a1672f90beed\") "
Oct 29 23:06:20.331431 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.331401    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb7c22d-bbfc-4ade-ab22-db401b342897-kube-api-access-vqzzg\" (UniqueName: \"kubernetes.io/projected/fbb7c22d-bbfc-4ade-ab22-db401b342897-kube-api-access-vqzzg\") pod \"fbb7c22d-bbfc-4ade-ab22-db401b342897\" (UID: \"fbb7c22d-bbfc-4ade-ab22-db401b342897\") "
Oct 29 23:06:20.331731 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.331698    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af142501-4ba8-4bb2-a7ce-a1672f90beed-plugins-config-volume" pod "af142501-4ba8-4bb2-a7ce-a1672f90beed" (UID: "af142501-4ba8-4bb2-a7ce-a1672f90beed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:06:20.331829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.331710    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb7c22d-bbfc-4ade-ab22-db401b342897-plugins-config-volume" pod "fbb7c22d-bbfc-4ade-ab22-db401b342897" (UID: "fbb7c22d-bbfc-4ade-ab22-db401b342897"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:06:20.331998 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.331977    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb7c22d-bbfc-4ade-ab22-db401b342897-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:06:20.332085 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.331997    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af142501-4ba8-4bb2-a7ce-a1672f90beed-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:06:20.333460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.333433    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af142501-4ba8-4bb2-a7ce-a1672f90beed-kube-api-access-nn4kf" pod "af142501-4ba8-4bb2-a7ce-a1672f90beed" (UID: "af142501-4ba8-4bb2-a7ce-a1672f90beed"). InnerVolumeSpecName "kube-api-access-nn4kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:06:20.333722 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.333700    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb7c22d-bbfc-4ade-ab22-db401b342897-kube-api-access-vqzzg" pod "fbb7c22d-bbfc-4ade-ab22-db401b342897" (UID: "fbb7c22d-bbfc-4ade-ab22-db401b342897"). InnerVolumeSpecName "kube-api-access-vqzzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:06:20.434064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.433965    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqzzg\" (UniqueName: \"kubernetes.io/projected/fbb7c22d-bbfc-4ade-ab22-db401b342897-kube-api-access-vqzzg\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:06:20.434064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.434002    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn4kf\" (UniqueName: \"kubernetes.io/projected/af142501-4ba8-4bb2-a7ce-a1672f90beed-kube-api-access-nn4kf\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:06:20.975709 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.975676    5428 generic.go:356] "Generic (PLEG): container finished" podID="fbb7c22d-bbfc-4ade-ab22-db401b342897" containerID="d1c59ea676f1602d80b144bf6ff7bcf89327e8707e0206c8050103ff7604e594" exitCode=0
Oct 29 23:06:20.975709 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.975721    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb"
Oct 29 23:06:20.975974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.975751    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb" event={"ID":"fbb7c22d-bbfc-4ade-ab22-db401b342897","Type":"ContainerDied","Data":"d1c59ea676f1602d80b144bf6ff7bcf89327e8707e0206c8050103ff7604e594"}
Oct 29 23:06:20.975974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.975832    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb" event={"ID":"fbb7c22d-bbfc-4ade-ab22-db401b342897","Type":"ContainerDied","Data":"3231f18039b4f0c79547109497abbd21879bc212b9ba2be6db4d8cb0bbef1944"}
Oct 29 23:06:20.975974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.975861    5428 scope.go:122] "RemoveContainer" containerID="d1c59ea676f1602d80b144bf6ff7bcf89327e8707e0206c8050103ff7604e594"
Oct 29 23:06:20.977637 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.977612    5428 generic.go:356] "Generic (PLEG): container finished" podID="af142501-4ba8-4bb2-a7ce-a1672f90beed" containerID="2314dfb20525de4cc85088333d6e0617f22d546c95404685afa2ec0125632957" exitCode=0
Oct 29 23:06:20.977749 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.977674    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j"
Oct 29 23:06:20.977983 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.977885    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j" event={"ID":"af142501-4ba8-4bb2-a7ce-a1672f90beed","Type":"ContainerDied","Data":"2314dfb20525de4cc85088333d6e0617f22d546c95404685afa2ec0125632957"}
Oct 29 23:06:20.977983 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.977940    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j" event={"ID":"af142501-4ba8-4bb2-a7ce-a1672f90beed","Type":"ContainerDied","Data":"234d148df61ed933e2da1c7c5d027b74194afacd3ed1b3e4436b26dd86194a81"}
Oct 29 23:06:20.987013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.986960    5428 scope.go:122] "RemoveContainer" containerID="d1c59ea676f1602d80b144bf6ff7bcf89327e8707e0206c8050103ff7604e594"
Oct 29 23:06:20.987248 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:06:20.987229    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c59ea676f1602d80b144bf6ff7bcf89327e8707e0206c8050103ff7604e594\": container with ID starting with d1c59ea676f1602d80b144bf6ff7bcf89327e8707e0206c8050103ff7604e594 not found: ID does not exist" containerID="d1c59ea676f1602d80b144bf6ff7bcf89327e8707e0206c8050103ff7604e594"
Oct 29 23:06:20.987310 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.987257    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c59ea676f1602d80b144bf6ff7bcf89327e8707e0206c8050103ff7604e594"} err="failed to get container status \"d1c59ea676f1602d80b144bf6ff7bcf89327e8707e0206c8050103ff7604e594\": rpc error: code = NotFound desc = could not find container \"d1c59ea676f1602d80b144bf6ff7bcf89327e8707e0206c8050103ff7604e594\": container with ID starting with d1c59ea676f1602d80b144bf6ff7bcf89327e8707e0206c8050103ff7604e594 not found: ID does not exist"
Oct 29 23:06:20.987310 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.987282    5428 scope.go:122] "RemoveContainer" containerID="2314dfb20525de4cc85088333d6e0617f22d546c95404685afa2ec0125632957"
Oct 29 23:06:20.996763 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.996744    5428 scope.go:122] "RemoveContainer" containerID="2314dfb20525de4cc85088333d6e0617f22d546c95404685afa2ec0125632957"
Oct 29 23:06:20.997016 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:06:20.996999    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2314dfb20525de4cc85088333d6e0617f22d546c95404685afa2ec0125632957\": container with ID starting with 2314dfb20525de4cc85088333d6e0617f22d546c95404685afa2ec0125632957 not found: ID does not exist" containerID="2314dfb20525de4cc85088333d6e0617f22d546c95404685afa2ec0125632957"
Oct 29 23:06:20.997082 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:20.997022    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2314dfb20525de4cc85088333d6e0617f22d546c95404685afa2ec0125632957"} err="failed to get container status \"2314dfb20525de4cc85088333d6e0617f22d546c95404685afa2ec0125632957\": rpc error: code = NotFound desc = could not find container \"2314dfb20525de4cc85088333d6e0617f22d546c95404685afa2ec0125632957\": container with ID starting with 2314dfb20525de4cc85088333d6e0617f22d546c95404685afa2ec0125632957 not found: ID does not exist"
Oct 29 23:06:21.040499 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:21.040419    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j"]
Oct 29 23:06:21.040499 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:21.040494    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-618d6c-6f6594fc55-nvr2j"]
Oct 29 23:06:21.067627 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:21.067563    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb"]
Oct 29 23:06:21.071617 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:21.071594    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-618d6c-6854cf85d6-kjwhb"]
Oct 29 23:06:21.543758 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:21.543726    5428 scope.go:122] "RemoveContainer" containerID="878f7d3c6c2646f3c7b3e350c7219287431b0e6fa8f19435433c51ced6a444eb"
Oct 29 23:06:21.686729 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:21.686695    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:22.779067 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:22.779034    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af142501-4ba8-4bb2-a7ce-a1672f90beed" path="/var/lib/kubelet/pods/af142501-4ba8-4bb2-a7ce-a1672f90beed/volumes"
Oct 29 23:06:22.779499 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:22.779484    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb7c22d-bbfc-4ade-ab22-db401b342897" path="/var/lib/kubelet/pods/fbb7c22d-bbfc-4ade-ab22-db401b342897/volumes"
Oct 29 23:06:23.495781 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:23.495751    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:23.910917 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:23.910887    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2"
Oct 29 23:06:25.621010 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:25.620974    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:26.412214 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:26.412179    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-822k7_5d767414-e11d-4113-90ee-f2f0ae1588a1/payload-processing/0.log"
Oct 29 23:06:26.464486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:26.464441    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:26.629415 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:26.629389    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-822k7_5d767414-e11d-4113-90ee-f2f0ae1588a1/payload-processing/0.log"
Oct 29 23:06:27.677217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:27.677181    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:28.003666 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:28.003571    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:28.121617 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:28.121585    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:29.490004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:29.489975    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:30.065032 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:30.065004    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:31.833688 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:31.833654    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:34.997592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:34.997558    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:37.270571 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:37.270535    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:39.426214 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:39.426173    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:40.128135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:40.128104    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:41.352781 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:41.352754    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:41.681299 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:41.681263    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj"]
Oct 29 23:06:41.712095 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:41.712066    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls"]
Oct 29 23:06:41.846751 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:41.846721    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:42.000877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:41.999124    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:42.154512 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:42.154474    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:42.296702 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:42.296614    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:42.437135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:42.437086    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:43.534553 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:43.534521    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:43.997857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:43.997824    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:44.204875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:44.204843    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:44.350107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:44.350025    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:44.487206 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:44.487169    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:45.127288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.127252    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"]
Oct 29 23:06:45.128095 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.128075    5428 cpu_manager.go:414] "containerMap: removing container" podUID="fbb7c22d-bbfc-4ade-ab22-db401b342897" containerName="payload-processing"
Oct 29 23:06:45.128175 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.128095    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb7c22d-bbfc-4ade-ab22-db401b342897" containerName="payload-processing"
Oct 29 23:06:45.128175 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.128107    5428 cpu_manager.go:414] "containerMap: removing container" podUID="af142501-4ba8-4bb2-a7ce-a1672f90beed" containerName="payload-pre-processing"
Oct 29 23:06:45.128175 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.128113    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af142501-4ba8-4bb2-a7ce-a1672f90beed" containerName="payload-pre-processing"
Oct 29 23:06:45.128331 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.128237    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb7c22d-bbfc-4ade-ab22-db401b342897" containerName="payload-processing"
Oct 29 23:06:45.128331 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.128246    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="af142501-4ba8-4bb2-a7ce-a1672f90beed" containerName="payload-pre-processing"
Oct 29 23:06:45.132117 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.132097    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"
Oct 29 23:06:45.151855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.151831    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-921680-serving-cert\""
Oct 29 23:06:45.170314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.170283    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"]
Oct 29 23:06:45.229415 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.229375    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25q7z\" (UniqueName: \"kubernetes.io/projected/0ff6b960-7094-4c65-b7d8-17bae2645ccb-kube-api-access-25q7z\") pod \"maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv\" (UID: \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"
Oct 29 23:06:45.229597 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.229424    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff6b960-7094-4c65-b7d8-17bae2645ccb-metrics-certs\") pod \"maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv\" (UID: \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"
Oct 29 23:06:45.229597 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.229544    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ff6b960-7094-4c65-b7d8-17bae2645ccb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv\" (UID: \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"
Oct 29 23:06:45.229707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.229650    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ff6b960-7094-4c65-b7d8-17bae2645ccb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv\" (UID: \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"
Oct 29 23:06:45.239859 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.239830    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:45.330975 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.330939    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25q7z\" (UniqueName: \"kubernetes.io/projected/0ff6b960-7094-4c65-b7d8-17bae2645ccb-kube-api-access-25q7z\") pod \"maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv\" (UID: \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"
Oct 29 23:06:45.331160 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.330985    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff6b960-7094-4c65-b7d8-17bae2645ccb-metrics-certs\") pod \"maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv\" (UID: \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"
Oct 29 23:06:45.331160 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.331040    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ff6b960-7094-4c65-b7d8-17bae2645ccb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv\" (UID: \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"
Oct 29 23:06:45.331160 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.331096    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ff6b960-7094-4c65-b7d8-17bae2645ccb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv\" (UID: \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"
Oct 29 23:06:45.331775 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.331751    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ff6b960-7094-4c65-b7d8-17bae2645ccb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv\" (UID: \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"
Oct 29 23:06:45.333605 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.333585    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff6b960-7094-4c65-b7d8-17bae2645ccb-metrics-certs\") pod \"maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv\" (UID: \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"
Oct 29 23:06:45.333729 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.333628    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ff6b960-7094-4c65-b7d8-17bae2645ccb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv\" (UID: \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"
Oct 29 23:06:45.354215 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.354188    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25q7z\" (UniqueName: \"kubernetes.io/projected/0ff6b960-7094-4c65-b7d8-17bae2645ccb-kube-api-access-25q7z\") pod \"maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv\" (UID: \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"
Oct 29 23:06:45.359248 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.359221    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:45.449124 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.449025    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"
Oct 29 23:06:45.468792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.468755    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:45.776039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:45.776008    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"]
Oct 29 23:06:45.777091 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:06:45.777064    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff6b960_7094_4c65_b7d8_17bae2645ccb.slice/crio-d992a4c88efa0fed6ddba0fb126b4c5b07ed185e52819c01d631cb76f429f623 WatchSource:0}: Error finding container d992a4c88efa0fed6ddba0fb126b4c5b07ed185e52819c01d631cb76f429f623: Status 404 returned error can't find the container with id d992a4c88efa0fed6ddba0fb126b4c5b07ed185e52819c01d631cb76f429f623
Oct 29 23:06:46.216443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:46.216398    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv" event={"ID":"0ff6b960-7094-4c65-b7d8-17bae2645ccb","Type":"ContainerStarted","Data":"d992a4c88efa0fed6ddba0fb126b4c5b07ed185e52819c01d631cb76f429f623"}
Oct 29 23:06:46.682069 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:46.682029    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj" podUID="a4ec35bb-8ae9-4800-87a9-70907abc1a09" containerName="payload-processing" containerID="cri-o://03cbae06db73c04c388648d020f19234404097a0eee24a33253da6dc7cc238f1" gracePeriod=25
Oct 29 23:06:46.712881 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:46.712845    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls" podUID="67b3bd3b-224d-47b3-aee2-145b94b2635b" containerName="payload-pre-processing" containerID="cri-o://7b069b9b85c42ef5348b95593627c3ca44a0a2d21685f5061106ef4bb3e11c74" gracePeriod=25
Oct 29 23:06:47.014643 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.014606    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj"
Oct 29 23:06:47.041591 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.041567    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls"
Oct 29 23:06:47.059734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.059688    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ec35bb-8ae9-4800-87a9-70907abc1a09-kube-api-access-zk2xv\" (UniqueName: \"kubernetes.io/projected/a4ec35bb-8ae9-4800-87a9-70907abc1a09-kube-api-access-zk2xv\") pod \"a4ec35bb-8ae9-4800-87a9-70907abc1a09\" (UID: \"a4ec35bb-8ae9-4800-87a9-70907abc1a09\") "
Oct 29 23:06:47.065300 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.065269    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4ec35bb-8ae9-4800-87a9-70907abc1a09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4ec35bb-8ae9-4800-87a9-70907abc1a09-plugins-config-volume\") pod \"a4ec35bb-8ae9-4800-87a9-70907abc1a09\" (UID: \"a4ec35bb-8ae9-4800-87a9-70907abc1a09\") "
Oct 29 23:06:47.067086 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.067032    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:47.073499 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.073047    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ec35bb-8ae9-4800-87a9-70907abc1a09-kube-api-access-zk2xv" pod "a4ec35bb-8ae9-4800-87a9-70907abc1a09" (UID: "a4ec35bb-8ae9-4800-87a9-70907abc1a09"). InnerVolumeSpecName "kube-api-access-zk2xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:06:47.081871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.076783    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ec35bb-8ae9-4800-87a9-70907abc1a09-plugins-config-volume" pod "a4ec35bb-8ae9-4800-87a9-70907abc1a09" (UID: "a4ec35bb-8ae9-4800-87a9-70907abc1a09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:06:47.167133 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.167100    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b3bd3b-224d-47b3-aee2-145b94b2635b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b3bd3b-224d-47b3-aee2-145b94b2635b-plugins-config-volume\") pod \"67b3bd3b-224d-47b3-aee2-145b94b2635b\" (UID: \"67b3bd3b-224d-47b3-aee2-145b94b2635b\") "
Oct 29 23:06:47.167133 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.167151    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b3bd3b-224d-47b3-aee2-145b94b2635b-kube-api-access-9mr7c\" (UniqueName: \"kubernetes.io/projected/67b3bd3b-224d-47b3-aee2-145b94b2635b-kube-api-access-9mr7c\") pod \"67b3bd3b-224d-47b3-aee2-145b94b2635b\" (UID: \"67b3bd3b-224d-47b3-aee2-145b94b2635b\") "
Oct 29 23:06:47.167886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.167862    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4ec35bb-8ae9-4800-87a9-70907abc1a09-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:06:47.167886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.167886    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk2xv\" (UniqueName: \"kubernetes.io/projected/a4ec35bb-8ae9-4800-87a9-70907abc1a09-kube-api-access-zk2xv\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:06:47.168061 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.167919    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b3bd3b-224d-47b3-aee2-145b94b2635b-plugins-config-volume" pod "67b3bd3b-224d-47b3-aee2-145b94b2635b" (UID: "67b3bd3b-224d-47b3-aee2-145b94b2635b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:06:47.170567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.170543    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b3bd3b-224d-47b3-aee2-145b94b2635b-kube-api-access-9mr7c" pod "67b3bd3b-224d-47b3-aee2-145b94b2635b" (UID: "67b3bd3b-224d-47b3-aee2-145b94b2635b"). InnerVolumeSpecName "kube-api-access-9mr7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:06:47.224921 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.224855    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:47.226554 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.226530    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv" event={"ID":"0ff6b960-7094-4c65-b7d8-17bae2645ccb","Type":"ContainerStarted","Data":"f2293586bc5bbd9d75856714978e8b9d3ace9ed83beaa11df6476c3d2c00ed6a"}
Oct 29 23:06:47.226664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.226607    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"
Oct 29 23:06:47.228012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.227988    5428 generic.go:356] "Generic (PLEG): container finished" podID="a4ec35bb-8ae9-4800-87a9-70907abc1a09" containerID="03cbae06db73c04c388648d020f19234404097a0eee24a33253da6dc7cc238f1" exitCode=0
Oct 29 23:06:47.228157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.228024    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj"
Oct 29 23:06:47.228157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.228068    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj" event={"ID":"a4ec35bb-8ae9-4800-87a9-70907abc1a09","Type":"ContainerDied","Data":"03cbae06db73c04c388648d020f19234404097a0eee24a33253da6dc7cc238f1"}
Oct 29 23:06:47.228157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.228113    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj" event={"ID":"a4ec35bb-8ae9-4800-87a9-70907abc1a09","Type":"ContainerDied","Data":"836834baa08cc186162eb3102c98ee546cff816ebd88975296f6f70707109fd5"}
Oct 29 23:06:47.228157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.228143    5428 scope.go:122] "RemoveContainer" containerID="03cbae06db73c04c388648d020f19234404097a0eee24a33253da6dc7cc238f1"
Oct 29 23:06:47.229537 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.229517    5428 generic.go:356] "Generic (PLEG): container finished" podID="67b3bd3b-224d-47b3-aee2-145b94b2635b" containerID="7b069b9b85c42ef5348b95593627c3ca44a0a2d21685f5061106ef4bb3e11c74" exitCode=0
Oct 29 23:06:47.229621 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.229562    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls"
Oct 29 23:06:47.229621 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.229585    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls" event={"ID":"67b3bd3b-224d-47b3-aee2-145b94b2635b","Type":"ContainerDied","Data":"7b069b9b85c42ef5348b95593627c3ca44a0a2d21685f5061106ef4bb3e11c74"}
Oct 29 23:06:47.229735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.229622    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls" event={"ID":"67b3bd3b-224d-47b3-aee2-145b94b2635b","Type":"ContainerDied","Data":"e9458289cfcac60b919b1bd74ef278d6a0348c5861f085cdea887b43cd280af7"}
Oct 29 23:06:47.240141 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.240126    5428 scope.go:122] "RemoveContainer" containerID="03cbae06db73c04c388648d020f19234404097a0eee24a33253da6dc7cc238f1"
Oct 29 23:06:47.240492 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:06:47.240473    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03cbae06db73c04c388648d020f19234404097a0eee24a33253da6dc7cc238f1\": container with ID starting with 03cbae06db73c04c388648d020f19234404097a0eee24a33253da6dc7cc238f1 not found: ID does not exist" containerID="03cbae06db73c04c388648d020f19234404097a0eee24a33253da6dc7cc238f1"
Oct 29 23:06:47.240563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.240499    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03cbae06db73c04c388648d020f19234404097a0eee24a33253da6dc7cc238f1"} err="failed to get container status \"03cbae06db73c04c388648d020f19234404097a0eee24a33253da6dc7cc238f1\": rpc error: code = NotFound desc = could not find container \"03cbae06db73c04c388648d020f19234404097a0eee24a33253da6dc7cc238f1\": container with ID starting with 03cbae06db73c04c388648d020f19234404097a0eee24a33253da6dc7cc238f1 not found: ID does not exist"
Oct 29 23:06:47.240563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.240518    5428 scope.go:122] "RemoveContainer" containerID="7b069b9b85c42ef5348b95593627c3ca44a0a2d21685f5061106ef4bb3e11c74"
Oct 29 23:06:47.251490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.251472    5428 scope.go:122] "RemoveContainer" containerID="7b069b9b85c42ef5348b95593627c3ca44a0a2d21685f5061106ef4bb3e11c74"
Oct 29 23:06:47.251757 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:06:47.251735    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b069b9b85c42ef5348b95593627c3ca44a0a2d21685f5061106ef4bb3e11c74\": container with ID starting with 7b069b9b85c42ef5348b95593627c3ca44a0a2d21685f5061106ef4bb3e11c74 not found: ID does not exist" containerID="7b069b9b85c42ef5348b95593627c3ca44a0a2d21685f5061106ef4bb3e11c74"
Oct 29 23:06:47.251852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.251763    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b069b9b85c42ef5348b95593627c3ca44a0a2d21685f5061106ef4bb3e11c74"} err="failed to get container status \"7b069b9b85c42ef5348b95593627c3ca44a0a2d21685f5061106ef4bb3e11c74\": rpc error: code = NotFound desc = could not find container \"7b069b9b85c42ef5348b95593627c3ca44a0a2d21685f5061106ef4bb3e11c74\": container with ID starting with 7b069b9b85c42ef5348b95593627c3ca44a0a2d21685f5061106ef4bb3e11c74 not found: ID does not exist"
Oct 29 23:06:47.269630 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.269610    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b3bd3b-224d-47b3-aee2-145b94b2635b-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:06:47.269630 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.269634    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mr7c\" (UniqueName: \"kubernetes.io/projected/67b3bd3b-224d-47b3-aee2-145b94b2635b-kube-api-access-9mr7c\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:06:47.283257 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.283216    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv" podStartSLOduration=1.8597987599999999 podStartE2EDuration="2.283202496s" totalImagesPullingTime="423.403736ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:47.281287219 +0000 UTC m=+1410.624235964" watchObservedRunningTime="2026-10-29 23:06:47.283202496 +0000 UTC m=+1410.626151244"
Oct 29 23:06:47.324670 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.324641    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj"]
Oct 29 23:06:47.333665 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.333631    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0c72c-56fb9f99dd-5qrrj"]
Oct 29 23:06:47.366457 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.366424    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls"]
Oct 29 23:06:47.371182 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.371158    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls"]
Oct 29 23:06:47.371921 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:06:47.371888    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:06:47.654360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:47.654327    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:48.024868 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:48.024753    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0c72c-6d7b9fd6cf-vbxls" podUID="67b3bd3b-224d-47b3-aee2-145b94b2635b" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.132:50052\" within 1s: context deadline exceeded"
Oct 29 23:06:48.025016 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:06:48.024889    5428 logging.go:55] [core] [Channel #2438 SubChannel #2439]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.132:50052", ServerName: "10.133.0.132:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.132:50052: operation was canceled"
Oct 29 23:06:48.690623 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:48.690580    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:48.780409 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:48.780378    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b3bd3b-224d-47b3-aee2-145b94b2635b" path="/var/lib/kubelet/pods/67b3bd3b-224d-47b3-aee2-145b94b2635b/volumes"
Oct 29 23:06:48.780946 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:48.780931    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4ec35bb-8ae9-4800-87a9-70907abc1a09" path="/var/lib/kubelet/pods/a4ec35bb-8ae9-4800-87a9-70907abc1a09/volumes"
Oct 29 23:06:48.846424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:48.846383    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:48.922827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:48.922547    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f"]
Oct 29 23:06:48.923951 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:48.923909    5428 cpu_manager.go:414] "containerMap: removing container" podUID="a4ec35bb-8ae9-4800-87a9-70907abc1a09" containerName="payload-processing"
Oct 29 23:06:48.923951 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:48.923950    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4ec35bb-8ae9-4800-87a9-70907abc1a09" containerName="payload-processing"
Oct 29 23:06:48.924183 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:48.924031    5428 cpu_manager.go:414] "containerMap: removing container" podUID="67b3bd3b-224d-47b3-aee2-145b94b2635b" containerName="payload-pre-processing"
Oct 29 23:06:48.924183 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:48.924042    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b3bd3b-224d-47b3-aee2-145b94b2635b" containerName="payload-pre-processing"
Oct 29 23:06:48.924261 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:48.924214    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4ec35bb-8ae9-4800-87a9-70907abc1a09" containerName="payload-processing"
Oct 29 23:06:48.924261 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:48.924241    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b3bd3b-224d-47b3-aee2-145b94b2635b" containerName="payload-pre-processing"
Oct 29 23:06:48.930378 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:48.930356    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f"
Oct 29 23:06:48.968774 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:48.957650    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f"]
Oct 29 23:06:48.968774 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:48.964010    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-cd000a-dockercfg-s9hqs\""
Oct 29 23:06:48.975011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:48.974970    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-cd000a\""
Oct 29 23:06:49.008283 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:49.008236    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b78432-161a-418b-aeaf-a0580dc80b54-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f\" (UID: \"f8b78432-161a-418b-aeaf-a0580dc80b54\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f"
Oct 29 23:06:49.008485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:49.008320    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59xs8\" (UniqueName: \"kubernetes.io/projected/f8b78432-161a-418b-aeaf-a0580dc80b54-kube-api-access-59xs8\") pod \"payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f\" (UID: \"f8b78432-161a-418b-aeaf-a0580dc80b54\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f"
Oct 29 23:06:49.110828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:49.110467    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b78432-161a-418b-aeaf-a0580dc80b54-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f\" (UID: \"f8b78432-161a-418b-aeaf-a0580dc80b54\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f"
Oct 29 23:06:49.110828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:49.110542    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59xs8\" (UniqueName: \"kubernetes.io/projected/f8b78432-161a-418b-aeaf-a0580dc80b54-kube-api-access-59xs8\") pod \"payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f\" (UID: \"f8b78432-161a-418b-aeaf-a0580dc80b54\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f"
Oct 29 23:06:49.117312 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:49.111825    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b78432-161a-418b-aeaf-a0580dc80b54-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f\" (UID: \"f8b78432-161a-418b-aeaf-a0580dc80b54\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f"
Oct 29 23:06:49.133915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:49.133881    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59xs8\" (UniqueName: \"kubernetes.io/projected/f8b78432-161a-418b-aeaf-a0580dc80b54-kube-api-access-59xs8\") pod \"payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f\" (UID: \"f8b78432-161a-418b-aeaf-a0580dc80b54\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f"
Oct 29 23:06:49.246050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:49.245956    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f"
Oct 29 23:06:49.565956 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:49.565929    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f"]
Oct 29 23:06:49.566399 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:06:49.566372    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b78432_161a_418b_aeaf_a0580dc80b54.slice/crio-3c3f7a3496a00f3815c164a8ca5169d8fd8a270fb1e3dd3610a35070e14976d2 WatchSource:0}: Error finding container 3c3f7a3496a00f3815c164a8ca5169d8fd8a270fb1e3dd3610a35070e14976d2: Status 404 returned error can't find the container with id 3c3f7a3496a00f3815c164a8ca5169d8fd8a270fb1e3dd3610a35070e14976d2
Oct 29 23:06:50.263730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.263686    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f" event={"ID":"f8b78432-161a-418b-aeaf-a0580dc80b54","Type":"ContainerStarted","Data":"f5af08cccd903e72b518a5a66a675f7564afd562c1626719c58d78d474e5b61d"}
Oct 29 23:06:50.264236 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.263750    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f" event={"ID":"f8b78432-161a-418b-aeaf-a0580dc80b54","Type":"ContainerStarted","Data":"3c3f7a3496a00f3815c164a8ca5169d8fd8a270fb1e3dd3610a35070e14976d2"}
Oct 29 23:06:50.264236 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.263826    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f"
Oct 29 23:06:50.313679 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.313647    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:50.399767 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.399698    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f" podStartSLOduration=2.399675513 podStartE2EDuration="2.399675513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:50.314107843 +0000 UTC m=+1413.657056593" watchObservedRunningTime="2026-10-29 23:06:50.399675513 +0000 UTC m=+1413.742624259"
Oct 29 23:06:50.403290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.403261    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98"]
Oct 29 23:06:50.407344 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.407324    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98"
Oct 29 23:06:50.422074 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.422053    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-c23236\""
Oct 29 23:06:50.422241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.422123    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-c23236-dockercfg-nt8gj\""
Oct 29 23:06:50.426288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.426261    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:50.427548 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.427526    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98"]
Oct 29 23:06:50.497924 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.497890    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z"]
Oct 29 23:06:50.513339 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.513295    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z"]
Oct 29 23:06:50.513540 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.513444    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z"
Oct 29 23:06:50.548671 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.548608    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb8qh\" (UniqueName: \"kubernetes.io/projected/fa734e34-458a-4633-9caf-c1e1b97f5c6f-kube-api-access-rb8qh\") pod \"payload-processing-e2e-praxis-b-c23236-6f96554868-crh98\" (UID: \"fa734e34-458a-4633-9caf-c1e1b97f5c6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98"
Oct 29 23:06:50.549726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.549699    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa734e34-458a-4633-9caf-c1e1b97f5c6f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c23236-6f96554868-crh98\" (UID: \"fa734e34-458a-4633-9caf-c1e1b97f5c6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98"
Oct 29 23:06:50.652844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.652789    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02f4bd43-6a70-471e-aa6c-459bdeaca665-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z\" (UID: \"02f4bd43-6a70-471e-aa6c-459bdeaca665\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z"
Oct 29 23:06:50.653210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.653170    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb8qh\" (UniqueName: \"kubernetes.io/projected/fa734e34-458a-4633-9caf-c1e1b97f5c6f-kube-api-access-rb8qh\") pod \"payload-processing-e2e-praxis-b-c23236-6f96554868-crh98\" (UID: \"fa734e34-458a-4633-9caf-c1e1b97f5c6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98"
Oct 29 23:06:50.653384 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.653363    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa734e34-458a-4633-9caf-c1e1b97f5c6f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c23236-6f96554868-crh98\" (UID: \"fa734e34-458a-4633-9caf-c1e1b97f5c6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98"
Oct 29 23:06:50.653454 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.653443    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhlc8\" (UniqueName: \"kubernetes.io/projected/02f4bd43-6a70-471e-aa6c-459bdeaca665-kube-api-access-zhlc8\") pod \"payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z\" (UID: \"02f4bd43-6a70-471e-aa6c-459bdeaca665\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z"
Oct 29 23:06:50.654770 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.654742    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa734e34-458a-4633-9caf-c1e1b97f5c6f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c23236-6f96554868-crh98\" (UID: \"fa734e34-458a-4633-9caf-c1e1b97f5c6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98"
Oct 29 23:06:50.677563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.677532    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb8qh\" (UniqueName: \"kubernetes.io/projected/fa734e34-458a-4633-9caf-c1e1b97f5c6f-kube-api-access-rb8qh\") pod \"payload-processing-e2e-praxis-b-c23236-6f96554868-crh98\" (UID: \"fa734e34-458a-4633-9caf-c1e1b97f5c6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98"
Oct 29 23:06:50.724937 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.724903    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98"
Oct 29 23:06:50.754463 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.754428    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhlc8\" (UniqueName: \"kubernetes.io/projected/02f4bd43-6a70-471e-aa6c-459bdeaca665-kube-api-access-zhlc8\") pod \"payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z\" (UID: \"02f4bd43-6a70-471e-aa6c-459bdeaca665\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z"
Oct 29 23:06:50.754602 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.754548    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02f4bd43-6a70-471e-aa6c-459bdeaca665-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z\" (UID: \"02f4bd43-6a70-471e-aa6c-459bdeaca665\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z"
Oct 29 23:06:50.755333 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.755291    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02f4bd43-6a70-471e-aa6c-459bdeaca665-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z\" (UID: \"02f4bd43-6a70-471e-aa6c-459bdeaca665\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z"
Oct 29 23:06:50.781141 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.781098    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhlc8\" (UniqueName: \"kubernetes.io/projected/02f4bd43-6a70-471e-aa6c-459bdeaca665-kube-api-access-zhlc8\") pod \"payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z\" (UID: \"02f4bd43-6a70-471e-aa6c-459bdeaca665\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z"
Oct 29 23:06:50.940942 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:50.940904    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z"
Oct 29 23:06:51.049896 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:51.049704    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98"]
Oct 29 23:06:51.054613 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:06:51.054582    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa734e34_458a_4633_9caf_c1e1b97f5c6f.slice/crio-e75b5461aa6346255d1f29c3e65219c6b49ac5ef218a2397eb1cd7c9e7f0c017 WatchSource:0}: Error finding container e75b5461aa6346255d1f29c3e65219c6b49ac5ef218a2397eb1cd7c9e7f0c017: Status 404 returned error can't find the container with id e75b5461aa6346255d1f29c3e65219c6b49ac5ef218a2397eb1cd7c9e7f0c017
Oct 29 23:06:51.273483 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:51.273446    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98" event={"ID":"fa734e34-458a-4633-9caf-c1e1b97f5c6f","Type":"ContainerStarted","Data":"5f87f4332bdc78c191aecf3ac6cf77c6090decc442e7c010e619e823409b827a"}
Oct 29 23:06:51.274012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:51.273512    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98" event={"ID":"fa734e34-458a-4633-9caf-c1e1b97f5c6f","Type":"ContainerStarted","Data":"e75b5461aa6346255d1f29c3e65219c6b49ac5ef218a2397eb1cd7c9e7f0c017"}
Oct 29 23:06:51.274012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:51.273543    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98"
Oct 29 23:06:51.314597 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:06:51.314569    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f4bd43_6a70_471e_aa6c_459bdeaca665.slice/crio-fd78681f7a087bf580c0732f98887b040383d929f3f1ba8f7cbf455a3f34f1f9 WatchSource:0}: Error finding container fd78681f7a087bf580c0732f98887b040383d929f3f1ba8f7cbf455a3f34f1f9: Status 404 returned error can't find the container with id fd78681f7a087bf580c0732f98887b040383d929f3f1ba8f7cbf455a3f34f1f9
Oct 29 23:06:51.318781 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:51.318753    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z"]
Oct 29 23:06:51.335950 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:51.335902    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98" podStartSLOduration=1.335887678 podStartE2EDuration="1.335887678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:51.333115556 +0000 UTC m=+1414.676064328" watchObservedRunningTime="2026-10-29 23:06:51.335887678 +0000 UTC m=+1414.678836422"
Oct 29 23:06:52.045057 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.045016    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"]
Oct 29 23:06:52.049705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.049689    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"
Oct 29 23:06:52.069095 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.069063    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-957eef-serving-cert\""
Oct 29 23:06:52.073928 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.073908    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"]
Oct 29 23:06:52.146434 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.146390    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:52.176494 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.176464    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2753c078-4e19-40c1-b4f0-c1591c6d64d4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk\" (UID: \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"
Oct 29 23:06:52.176647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.176610    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2753c078-4e19-40c1-b4f0-c1591c6d64d4-metrics-certs\") pod \"maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk\" (UID: \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"
Oct 29 23:06:52.176722 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.176647    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b7c7\" (UniqueName: \"kubernetes.io/projected/2753c078-4e19-40c1-b4f0-c1591c6d64d4-kube-api-access-5b7c7\") pod \"maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk\" (UID: \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"
Oct 29 23:06:52.176722 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.176669    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2753c078-4e19-40c1-b4f0-c1591c6d64d4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk\" (UID: \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"
Oct 29 23:06:52.265887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.265796    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:52.277958 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.277920    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2753c078-4e19-40c1-b4f0-c1591c6d64d4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk\" (UID: \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"
Oct 29 23:06:52.278403 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.278076    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2753c078-4e19-40c1-b4f0-c1591c6d64d4-metrics-certs\") pod \"maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk\" (UID: \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"
Oct 29 23:06:52.278403 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.278122    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b7c7\" (UniqueName: \"kubernetes.io/projected/2753c078-4e19-40c1-b4f0-c1591c6d64d4-kube-api-access-5b7c7\") pod \"maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk\" (UID: \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"
Oct 29 23:06:52.278403 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.278161    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2753c078-4e19-40c1-b4f0-c1591c6d64d4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk\" (UID: \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"
Oct 29 23:06:52.278744 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.278717    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2753c078-4e19-40c1-b4f0-c1591c6d64d4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk\" (UID: \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"
Oct 29 23:06:52.280793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.280766    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2753c078-4e19-40c1-b4f0-c1591c6d64d4-metrics-certs\") pod \"maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk\" (UID: \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"
Oct 29 23:06:52.280918 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.280772    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2753c078-4e19-40c1-b4f0-c1591c6d64d4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk\" (UID: \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"
Oct 29 23:06:52.283115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.283082    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z" event={"ID":"02f4bd43-6a70-471e-aa6c-459bdeaca665","Type":"ContainerStarted","Data":"b00eb59943c4265dc1de865f36186f3e3817c9d63b1fab108919301d376b40ae"}
Oct 29 23:06:52.283247 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.283150    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z" event={"ID":"02f4bd43-6a70-471e-aa6c-459bdeaca665","Type":"ContainerStarted","Data":"fd78681f7a087bf580c0732f98887b040383d929f3f1ba8f7cbf455a3f34f1f9"}
Oct 29 23:06:52.283247 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.283185    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z"
Oct 29 23:06:52.301522 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.301467    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b7c7\" (UniqueName: \"kubernetes.io/projected/2753c078-4e19-40c1-b4f0-c1591c6d64d4-kube-api-access-5b7c7\") pod \"maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk\" (UID: \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"
Oct 29 23:06:52.324646 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.324600    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z" podStartSLOduration=2.324586228 podStartE2EDuration="2.324586228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:52.323953778 +0000 UTC m=+1415.666902549" watchObservedRunningTime="2026-10-29 23:06:52.324586228 +0000 UTC m=+1415.667535000"
Oct 29 23:06:52.365544 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.365510    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"
Oct 29 23:06:52.382643 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.382618    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:52.684932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:52.684904    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"]
Oct 29 23:06:52.685657 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:06:52.685632    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2753c078_4e19_40c1_b4f0_c1591c6d64d4.slice/crio-65792026dbd4519a0be96465077869f5c6c1cbe45b8374048cbbd55b6cd5b50c WatchSource:0}: Error finding container 65792026dbd4519a0be96465077869f5c6c1cbe45b8374048cbbd55b6cd5b50c: Status 404 returned error can't find the container with id 65792026dbd4519a0be96465077869f5c6c1cbe45b8374048cbbd55b6cd5b50c
Oct 29 23:06:53.246474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:53.246445    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"
Oct 29 23:06:53.293043 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:53.293010    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk" event={"ID":"2753c078-4e19-40c1-b4f0-c1591c6d64d4","Type":"ContainerStarted","Data":"e305de0f1c917926a232c84fe67854c09e3c135468685c47687b981cfd685ba2"}
Oct 29 23:06:53.293490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:53.293069    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk" event={"ID":"2753c078-4e19-40c1-b4f0-c1591c6d64d4","Type":"ContainerStarted","Data":"65792026dbd4519a0be96465077869f5c6c1cbe45b8374048cbbd55b6cd5b50c"}
Oct 29 23:06:53.293490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:53.293097    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"
Oct 29 23:06:54.559967 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:54.559928    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:55.796057 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:55.796023    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:56.078013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:56.077932    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:57.605849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:57.605811    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:59.306625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:59.306597    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"
Oct 29 23:06:59.351371 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:59.351319    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk" podStartSLOduration=6.989454216 podStartE2EDuration="7.351305212s" totalImagesPullingTime="361.850996ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:53.343055323 +0000 UTC m=+1416.686004069" watchObservedRunningTime="2026-10-29 23:06:59.351305212 +0000 UTC m=+1422.694253994"
Oct 29 23:06:59.611608 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:59.611525    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:59.649426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:59.649387    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"]
Oct 29 23:06:59.649625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:59.649601    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt" podUID="7242d814-03a8-4079-9202-ede3445634b3" containerName="maas-api" containerID="cri-o://eb4ffe848a4dba35e05085e838ef329d7e3286f5a631593b2915c787f59613f5" gracePeriod=30
Oct 29 23:06:59.734065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:59.734033    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:59.877115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:59.874092    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:59.986046 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:59.986014    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:06:59.988794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:59.988768    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f"]
Oct 29 23:06:59.989035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:06:59.989002    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f" podUID="c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2" containerName="payload-processing" containerID="cri-o://c1ceb58dfdf80a91eee29c00d16810df3e0db3c285d76c1c9a903c24d12f4e20" gracePeriod=30
Oct 29 23:07:00.060189 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.060153    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d"]
Oct 29 23:07:00.060499 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.060433    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d" podUID="be26e3f8-a847-49bc-aa52-c37190c3c2c6" containerName="payload-pre-processing" containerID="cri-o://e2bc5c9cbc65f7157de8abb6c95f8a18d7d9f52a4eb847b379aca3520d6fee7b" gracePeriod=30
Oct 29 23:07:00.113625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.113595    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7242d814-03a8-4079-9202-ede3445634b3-kube-api-access-dwj87\" (UniqueName: \"kubernetes.io/projected/7242d814-03a8-4079-9202-ede3445634b3-kube-api-access-dwj87\") pod \"7242d814-03a8-4079-9202-ede3445634b3\" (UID: \"7242d814-03a8-4079-9202-ede3445634b3\") "
Oct 29 23:07:00.113790 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.113773    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7242d814-03a8-4079-9202-ede3445634b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7242d814-03a8-4079-9202-ede3445634b3-openshift-service-ca\") pod \"7242d814-03a8-4079-9202-ede3445634b3\" (UID: \"7242d814-03a8-4079-9202-ede3445634b3\") "
Oct 29 23:07:00.113915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.113898    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7242d814-03a8-4079-9202-ede3445634b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7242d814-03a8-4079-9202-ede3445634b3-maas-api-tls\") pod \"7242d814-03a8-4079-9202-ede3445634b3\" (UID: \"7242d814-03a8-4079-9202-ede3445634b3\") "
Oct 29 23:07:00.113981 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.113934    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7242d814-03a8-4079-9202-ede3445634b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7242d814-03a8-4079-9202-ede3445634b3-metrics-certs\") pod \"7242d814-03a8-4079-9202-ede3445634b3\" (UID: \"7242d814-03a8-4079-9202-ede3445634b3\") "
Oct 29 23:07:00.114294 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.114271    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7242d814-03a8-4079-9202-ede3445634b3-openshift-service-ca" pod "7242d814-03a8-4079-9202-ede3445634b3" (UID: "7242d814-03a8-4079-9202-ede3445634b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:00.114653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.114613    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7242d814-03a8-4079-9202-ede3445634b3-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:00.117034 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.117005    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7242d814-03a8-4079-9202-ede3445634b3-maas-api-tls" pod "7242d814-03a8-4079-9202-ede3445634b3" (UID: "7242d814-03a8-4079-9202-ede3445634b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:07:00.117181 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.117157    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7242d814-03a8-4079-9202-ede3445634b3-kube-api-access-dwj87" pod "7242d814-03a8-4079-9202-ede3445634b3" (UID: "7242d814-03a8-4079-9202-ede3445634b3"). InnerVolumeSpecName "kube-api-access-dwj87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:00.117305 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.117282    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7242d814-03a8-4079-9202-ede3445634b3-metrics-certs" pod "7242d814-03a8-4079-9202-ede3445634b3" (UID: "7242d814-03a8-4079-9202-ede3445634b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:07:00.128577 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.128512    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:00.216651 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.216570    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwj87\" (UniqueName: \"kubernetes.io/projected/7242d814-03a8-4079-9202-ede3445634b3-kube-api-access-dwj87\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:00.216651 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.216608    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7242d814-03a8-4079-9202-ede3445634b3-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:00.216651 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.216625    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7242d814-03a8-4079-9202-ede3445634b3-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:00.235430 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.235397    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"]
Oct 29 23:07:00.235840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.235810    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc" podUID="0748e57c-035e-47c3-a76f-bebe19de3139" containerName="istio-proxy" containerID="cri-o://2db79b7a37f4baf62119981bb9db223b7283b031802cc077d0929536a08427f5" gracePeriod=30
Oct 29 23:07:00.257828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.249725    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:00.340453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.340423    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f"
Oct 29 23:07:00.351788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.350152    5428 generic.go:356] "Generic (PLEG): container finished" podID="7242d814-03a8-4079-9202-ede3445634b3" containerID="eb4ffe848a4dba35e05085e838ef329d7e3286f5a631593b2915c787f59613f5" exitCode=0
Oct 29 23:07:00.351788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.350281    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt" event={"ID":"7242d814-03a8-4079-9202-ede3445634b3","Type":"ContainerDied","Data":"eb4ffe848a4dba35e05085e838ef329d7e3286f5a631593b2915c787f59613f5"}
Oct 29 23:07:00.351788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.350338    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt" event={"ID":"7242d814-03a8-4079-9202-ede3445634b3","Type":"ContainerDied","Data":"121e56c803dc8ab8464e2d3b3723f6facc6c7b8a79df86ecaeaaf0b3eaf2f4de"}
Oct 29 23:07:00.351788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.350373    5428 scope.go:122] "RemoveContainer" containerID="eb4ffe848a4dba35e05085e838ef329d7e3286f5a631593b2915c787f59613f5"
Oct 29 23:07:00.351788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.350604    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"
Oct 29 23:07:00.360726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.360534    5428 generic.go:356] "Generic (PLEG): container finished" podID="be26e3f8-a847-49bc-aa52-c37190c3c2c6" containerID="e2bc5c9cbc65f7157de8abb6c95f8a18d7d9f52a4eb847b379aca3520d6fee7b" exitCode=0
Oct 29 23:07:00.360726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.360647    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d" event={"ID":"be26e3f8-a847-49bc-aa52-c37190c3c2c6","Type":"ContainerDied","Data":"e2bc5c9cbc65f7157de8abb6c95f8a18d7d9f52a4eb847b379aca3520d6fee7b"}
Oct 29 23:07:00.363199 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.362863    5428 generic.go:356] "Generic (PLEG): container finished" podID="c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2" containerID="c1ceb58dfdf80a91eee29c00d16810df3e0db3c285d76c1c9a903c24d12f4e20" exitCode=0
Oct 29 23:07:00.363199 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.362920    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f" event={"ID":"c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2","Type":"ContainerDied","Data":"c1ceb58dfdf80a91eee29c00d16810df3e0db3c285d76c1c9a903c24d12f4e20"}
Oct 29 23:07:00.363199 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.362958    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f" event={"ID":"c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2","Type":"ContainerDied","Data":"1167827c92f76736c0789c04032030227016371acaa1fd71ff9b2b6dbf19e6d9"}
Oct 29 23:07:00.363199 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.363039    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f"
Oct 29 23:07:00.474824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.474778    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"]
Oct 29 23:07:00.483696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.481553    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-44a4e4-54b879b4c4-g5lnt"]
Oct 29 23:07:00.509726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.509481    5428 scope.go:122] "RemoveContainer" containerID="eb4ffe848a4dba35e05085e838ef329d7e3286f5a631593b2915c787f59613f5"
Oct 29 23:07:00.510156 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:07:00.509984    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4ffe848a4dba35e05085e838ef329d7e3286f5a631593b2915c787f59613f5\": container with ID starting with eb4ffe848a4dba35e05085e838ef329d7e3286f5a631593b2915c787f59613f5 not found: ID does not exist" containerID="eb4ffe848a4dba35e05085e838ef329d7e3286f5a631593b2915c787f59613f5"
Oct 29 23:07:00.510156 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.510023    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4ffe848a4dba35e05085e838ef329d7e3286f5a631593b2915c787f59613f5"} err="failed to get container status \"eb4ffe848a4dba35e05085e838ef329d7e3286f5a631593b2915c787f59613f5\": rpc error: code = NotFound desc = could not find container \"eb4ffe848a4dba35e05085e838ef329d7e3286f5a631593b2915c787f59613f5\": container with ID starting with eb4ffe848a4dba35e05085e838ef329d7e3286f5a631593b2915c787f59613f5 not found: ID does not exist"
Oct 29 23:07:00.510156 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.510051    5428 scope.go:122] "RemoveContainer" containerID="c1ceb58dfdf80a91eee29c00d16810df3e0db3c285d76c1c9a903c24d12f4e20"
Oct 29 23:07:00.522421 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.521414    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2-plugins-config-volume\") pod \"c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2\" (UID: \"c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2\") "
Oct 29 23:07:00.522421 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.521521    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2-kube-api-access-p6r5w\" (UniqueName: \"kubernetes.io/projected/c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2-kube-api-access-p6r5w\") pod \"c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2\" (UID: \"c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2\") "
Oct 29 23:07:00.522421 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.522360    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2-plugins-config-volume" pod "c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2" (UID: "c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:00.529721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.529695    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d"
Oct 29 23:07:00.534726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.534706    5428 scope.go:122] "RemoveContainer" containerID="c1ceb58dfdf80a91eee29c00d16810df3e0db3c285d76c1c9a903c24d12f4e20"
Oct 29 23:07:00.535065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.535036    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2-kube-api-access-p6r5w" pod "c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2" (UID: "c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2"). InnerVolumeSpecName "kube-api-access-p6r5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:00.535179 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:07:00.535083    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ceb58dfdf80a91eee29c00d16810df3e0db3c285d76c1c9a903c24d12f4e20\": container with ID starting with c1ceb58dfdf80a91eee29c00d16810df3e0db3c285d76c1c9a903c24d12f4e20 not found: ID does not exist" containerID="c1ceb58dfdf80a91eee29c00d16810df3e0db3c285d76c1c9a903c24d12f4e20"
Oct 29 23:07:00.535179 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.535108    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ceb58dfdf80a91eee29c00d16810df3e0db3c285d76c1c9a903c24d12f4e20"} err="failed to get container status \"c1ceb58dfdf80a91eee29c00d16810df3e0db3c285d76c1c9a903c24d12f4e20\": rpc error: code = NotFound desc = could not find container \"c1ceb58dfdf80a91eee29c00d16810df3e0db3c285d76c1c9a903c24d12f4e20\": container with ID starting with c1ceb58dfdf80a91eee29c00d16810df3e0db3c285d76c1c9a903c24d12f4e20 not found: ID does not exist"
Oct 29 23:07:00.622784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.622749    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be26e3f8-a847-49bc-aa52-c37190c3c2c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be26e3f8-a847-49bc-aa52-c37190c3c2c6-plugins-config-volume\") pod \"be26e3f8-a847-49bc-aa52-c37190c3c2c6\" (UID: \"be26e3f8-a847-49bc-aa52-c37190c3c2c6\") "
Oct 29 23:07:00.623036 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.623007    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be26e3f8-a847-49bc-aa52-c37190c3c2c6-kube-api-access-tfvhg\" (UniqueName: \"kubernetes.io/projected/be26e3f8-a847-49bc-aa52-c37190c3c2c6-kube-api-access-tfvhg\") pod \"be26e3f8-a847-49bc-aa52-c37190c3c2c6\" (UID: \"be26e3f8-a847-49bc-aa52-c37190c3c2c6\") "
Oct 29 23:07:00.623556 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.623527    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be26e3f8-a847-49bc-aa52-c37190c3c2c6-plugins-config-volume" pod "be26e3f8-a847-49bc-aa52-c37190c3c2c6" (UID: "be26e3f8-a847-49bc-aa52-c37190c3c2c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:00.623741 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.623722    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6r5w\" (UniqueName: \"kubernetes.io/projected/c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2-kube-api-access-p6r5w\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:00.623818 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.623746    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:00.623818 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.623763    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be26e3f8-a847-49bc-aa52-c37190c3c2c6-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:00.626483 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.626450    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be26e3f8-a847-49bc-aa52-c37190c3c2c6-kube-api-access-tfvhg" pod "be26e3f8-a847-49bc-aa52-c37190c3c2c6" (UID: "be26e3f8-a847-49bc-aa52-c37190c3c2c6"). InnerVolumeSpecName "kube-api-access-tfvhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:00.725752 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.725717    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfvhg\" (UniqueName: \"kubernetes.io/projected/be26e3f8-a847-49bc-aa52-c37190c3c2c6-kube-api-access-tfvhg\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:00.725999 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.725763    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f"]
Oct 29 23:07:00.729982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.729957    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-44a4e4-5489d67594-xz94f"]
Oct 29 23:07:00.787381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.787342    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7242d814-03a8-4079-9202-ede3445634b3" path="/var/lib/kubelet/pods/7242d814-03a8-4079-9202-ede3445634b3/volumes"
Oct 29 23:07:00.788040 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.788016    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2" path="/var/lib/kubelet/pods/c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2/volumes"
Oct 29 23:07:00.997622 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:00.997533    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:01.237408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:01.237371    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:01.277232 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:01.277135    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f"
Oct 29 23:07:01.375728 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:01.375695    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d" event={"ID":"be26e3f8-a847-49bc-aa52-c37190c3c2c6","Type":"ContainerDied","Data":"228c5d07ffd74392985334750967989eb6c34a0cca094cd3997a87b8d82dcf14"}
Oct 29 23:07:01.376213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:01.375750    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d"
Oct 29 23:07:01.376213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:01.375763    5428 scope.go:122] "RemoveContainer" containerID="e2bc5c9cbc65f7157de8abb6c95f8a18d7d9f52a4eb847b379aca3520d6fee7b"
Oct 29 23:07:01.432071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:01.432041    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d"]
Oct 29 23:07:01.436788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:01.436762    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-44a4e4-547589674c-j5z5d"]
Oct 29 23:07:01.477388 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:01.477348    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:01.832012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:01.831956    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:02.286924 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:02.286895    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98"
Oct 29 23:07:02.779491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:02.779461    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be26e3f8-a847-49bc-aa52-c37190c3c2c6" path="/var/lib/kubelet/pods/be26e3f8-a847-49bc-aa52-c37190c3c2c6/volumes"
Oct 29 23:07:03.296765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:03.296737    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z"
Oct 29 23:07:03.486685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:03.486658    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:05.342441 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.342408    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:05.419932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.419894    5428 generic.go:356] "Generic (PLEG): container finished" podID="0748e57c-035e-47c3-a76f-bebe19de3139" containerID="2db79b7a37f4baf62119981bb9db223b7283b031802cc077d0929536a08427f5" exitCode=0
Oct 29 23:07:05.420114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.419937    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc" event={"ID":"0748e57c-035e-47c3-a76f-bebe19de3139","Type":"ContainerDied","Data":"2db79b7a37f4baf62119981bb9db223b7283b031802cc077d0929536a08427f5"}
Oct 29 23:07:05.551199 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.551174    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 23:07:05.596334 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.596306    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-istio-data\") pod \"0748e57c-035e-47c3-a76f-bebe19de3139\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") "
Oct 29 23:07:05.596460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.596347    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-workload-certs\") pod \"0748e57c-035e-47c3-a76f-bebe19de3139\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") "
Oct 29 23:07:05.596460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.596420    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0748e57c-035e-47c3-a76f-bebe19de3139-istio-token\" (UniqueName: \"kubernetes.io/projected/0748e57c-035e-47c3-a76f-bebe19de3139-istio-token\") pod \"0748e57c-035e-47c3-a76f-bebe19de3139\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") "
Oct 29 23:07:05.596460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.596443    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0748e57c-035e-47c3-a76f-bebe19de3139-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0748e57c-035e-47c3-a76f-bebe19de3139-istio-podinfo\") pod \"0748e57c-035e-47c3-a76f-bebe19de3139\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") "
Oct 29 23:07:05.596594 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.596489    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0748e57c-035e-47c3-a76f-bebe19de3139-kube-api-access-swdkh\" (UniqueName: \"kubernetes.io/projected/0748e57c-035e-47c3-a76f-bebe19de3139-kube-api-access-swdkh\") pod \"0748e57c-035e-47c3-a76f-bebe19de3139\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") "
Oct 29 23:07:05.596594 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.596532    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-workload-socket\") pod \"0748e57c-035e-47c3-a76f-bebe19de3139\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") "
Oct 29 23:07:05.596689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.596603    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-istio-data" pod "0748e57c-035e-47c3-a76f-bebe19de3139" (UID: "0748e57c-035e-47c3-a76f-bebe19de3139"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:05.596689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.596634    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-workload-certs" pod "0748e57c-035e-47c3-a76f-bebe19de3139" (UID: "0748e57c-035e-47c3-a76f-bebe19de3139"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:05.596689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.596641    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-credential-socket\") pod \"0748e57c-035e-47c3-a76f-bebe19de3139\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") "
Oct 29 23:07:05.596871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.596742    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0748e57c-035e-47c3-a76f-bebe19de3139-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0748e57c-035e-47c3-a76f-bebe19de3139-istiod-ca-cert\") pod \"0748e57c-035e-47c3-a76f-bebe19de3139\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") "
Oct 29 23:07:05.596871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.596770    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-istio-envoy\") pod \"0748e57c-035e-47c3-a76f-bebe19de3139\" (UID: \"0748e57c-035e-47c3-a76f-bebe19de3139\") "
Oct 29 23:07:05.597075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.596893    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-workload-socket" pod "0748e57c-035e-47c3-a76f-bebe19de3139" (UID: "0748e57c-035e-47c3-a76f-bebe19de3139"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:05.597075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.596900    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-credential-socket" pod "0748e57c-035e-47c3-a76f-bebe19de3139" (UID: "0748e57c-035e-47c3-a76f-bebe19de3139"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:05.597163 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.597128    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0748e57c-035e-47c3-a76f-bebe19de3139-istiod-ca-cert" pod "0748e57c-035e-47c3-a76f-bebe19de3139" (UID: "0748e57c-035e-47c3-a76f-bebe19de3139"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:05.597456 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.597434    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:05.597456 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.597457    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0748e57c-035e-47c3-a76f-bebe19de3139-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:05.597610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.597471    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:05.597610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.597482    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:05.597610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.597494    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:05.598749 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.598725    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0748e57c-035e-47c3-a76f-bebe19de3139-istio-token" pod "0748e57c-035e-47c3-a76f-bebe19de3139" (UID: "0748e57c-035e-47c3-a76f-bebe19de3139"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:05.598903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.598888    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-istio-envoy" pod "0748e57c-035e-47c3-a76f-bebe19de3139" (UID: "0748e57c-035e-47c3-a76f-bebe19de3139"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:05.599213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.599196    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0748e57c-035e-47c3-a76f-bebe19de3139-kube-api-access-swdkh" pod "0748e57c-035e-47c3-a76f-bebe19de3139" (UID: "0748e57c-035e-47c3-a76f-bebe19de3139"). InnerVolumeSpecName "kube-api-access-swdkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:05.599306 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.599250    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0748e57c-035e-47c3-a76f-bebe19de3139-istio-podinfo" pod "0748e57c-035e-47c3-a76f-bebe19de3139" (UID: "0748e57c-035e-47c3-a76f-bebe19de3139"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:07:05.698664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.698633    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0748e57c-035e-47c3-a76f-bebe19de3139-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:05.698664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.698663    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0748e57c-035e-47c3-a76f-bebe19de3139-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:05.698664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.698673    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swdkh\" (UniqueName: \"kubernetes.io/projected/0748e57c-035e-47c3-a76f-bebe19de3139-kube-api-access-swdkh\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:05.698927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:05.698683    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0748e57c-035e-47c3-a76f-bebe19de3139-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:06.428433 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:06.428404    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"
Oct 29 23:07:06.428994 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:06.428407    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc" event={"ID":"0748e57c-035e-47c3-a76f-bebe19de3139","Type":"ContainerDied","Data":"5ba98790671be4401efe75773d19c786b6dfa59a6c93d1cf23034635b66503fd"}
Oct 29 23:07:06.428994 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:06.428534    5428 scope.go:122] "RemoveContainer" containerID="2db79b7a37f4baf62119981bb9db223b7283b031802cc077d0929536a08427f5"
Oct 29 23:07:06.494345 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:06.494316    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"]
Oct 29 23:07:06.506288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:06.506255    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-s8xkc"]
Oct 29 23:07:06.781556 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:06.781465    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0748e57c-035e-47c3-a76f-bebe19de3139" path="/var/lib/kubelet/pods/0748e57c-035e-47c3-a76f-bebe19de3139/volumes"
Oct 29 23:07:07.060233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:07.060144    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:08.665211 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:08.665174    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:08.794883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:08.794834    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:08.955280 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:08.955198    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:09.014070 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:09.011528    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"]
Oct 29 23:07:09.014070 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:09.011781    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp" podUID="cb016a04-d965-4ee7-b802-48c35f8c4b61" containerName="istio-proxy" containerID="cri-o://98f1b02998d9a24acb564d647d8a9e7403693d102e113562dad09020bb765cfc" gracePeriod=30
Oct 29 23:07:09.074085 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:09.074050    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:09.809712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:09.809678    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:11.695998 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:11.695958    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:13.878840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:13.878793    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:14.298363 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.298331    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:07:14.448372 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.448287    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-podinfo\") pod \"cb016a04-d965-4ee7-b802-48c35f8c4b61\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") "
Oct 29 23:07:14.448372 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.448331    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-workload-certs\") pod \"cb016a04-d965-4ee7-b802-48c35f8c4b61\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") "
Oct 29 23:07:14.448372 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.448356    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-data\") pod \"cb016a04-d965-4ee7-b802-48c35f8c4b61\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") "
Oct 29 23:07:14.448653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.448391    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-envoy\") pod \"cb016a04-d965-4ee7-b802-48c35f8c4b61\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") "
Oct 29 23:07:14.448653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.448466    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb016a04-d965-4ee7-b802-48c35f8c4b61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb016a04-d965-4ee7-b802-48c35f8c4b61-istiod-ca-cert\") pod \"cb016a04-d965-4ee7-b802-48c35f8c4b61\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") "
Oct 29 23:07:14.448653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.448546    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-credential-socket\") pod \"cb016a04-d965-4ee7-b802-48c35f8c4b61\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") "
Oct 29 23:07:14.448653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.448589    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb016a04-d965-4ee7-b802-48c35f8c4b61-kube-api-access-vhxbk\" (UniqueName: \"kubernetes.io/projected/cb016a04-d965-4ee7-b802-48c35f8c4b61-kube-api-access-vhxbk\") pod \"cb016a04-d965-4ee7-b802-48c35f8c4b61\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") "
Oct 29 23:07:14.448653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.448594    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-workload-certs" pod "cb016a04-d965-4ee7-b802-48c35f8c4b61" (UID: "cb016a04-d965-4ee7-b802-48c35f8c4b61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:14.448990 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.448675    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-workload-socket\") pod \"cb016a04-d965-4ee7-b802-48c35f8c4b61\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") "
Oct 29 23:07:14.448990 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.448724    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-token\" (UniqueName: \"kubernetes.io/projected/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-token\") pod \"cb016a04-d965-4ee7-b802-48c35f8c4b61\" (UID: \"cb016a04-d965-4ee7-b802-48c35f8c4b61\") "
Oct 29 23:07:14.448990 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.448757    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-data" pod "cb016a04-d965-4ee7-b802-48c35f8c4b61" (UID: "cb016a04-d965-4ee7-b802-48c35f8c4b61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:14.448990 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.448856    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-credential-socket" pod "cb016a04-d965-4ee7-b802-48c35f8c4b61" (UID: "cb016a04-d965-4ee7-b802-48c35f8c4b61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:14.448990 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.448904    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb016a04-d965-4ee7-b802-48c35f8c4b61-istiod-ca-cert" pod "cb016a04-d965-4ee7-b802-48c35f8c4b61" (UID: "cb016a04-d965-4ee7-b802-48c35f8c4b61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:14.449256 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.449023    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-workload-socket" pod "cb016a04-d965-4ee7-b802-48c35f8c4b61" (UID: "cb016a04-d965-4ee7-b802-48c35f8c4b61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:14.449572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.449551    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb016a04-d965-4ee7-b802-48c35f8c4b61-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:14.449572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.449572    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:14.449707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.449582    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:14.449707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.449592    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:14.449707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.449601    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:14.450648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.450623    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-podinfo" pod "cb016a04-d965-4ee7-b802-48c35f8c4b61" (UID: "cb016a04-d965-4ee7-b802-48c35f8c4b61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:07:14.450748 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.450714    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-envoy" pod "cb016a04-d965-4ee7-b802-48c35f8c4b61" (UID: "cb016a04-d965-4ee7-b802-48c35f8c4b61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:14.451192 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.451175    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-token" pod "cb016a04-d965-4ee7-b802-48c35f8c4b61" (UID: "cb016a04-d965-4ee7-b802-48c35f8c4b61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:14.451266 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.451201    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb016a04-d965-4ee7-b802-48c35f8c4b61-kube-api-access-vhxbk" pod "cb016a04-d965-4ee7-b802-48c35f8c4b61" (UID: "cb016a04-d965-4ee7-b802-48c35f8c4b61"). InnerVolumeSpecName "kube-api-access-vhxbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:14.500569 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.500536    5428 generic.go:356] "Generic (PLEG): container finished" podID="cb016a04-d965-4ee7-b802-48c35f8c4b61" containerID="98f1b02998d9a24acb564d647d8a9e7403693d102e113562dad09020bb765cfc" exitCode=0
Oct 29 23:07:14.500708 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.500587    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"
Oct 29 23:07:14.500708 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.500627    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp" event={"ID":"cb016a04-d965-4ee7-b802-48c35f8c4b61","Type":"ContainerDied","Data":"98f1b02998d9a24acb564d647d8a9e7403693d102e113562dad09020bb765cfc"}
Oct 29 23:07:14.500708 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.500696    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp" event={"ID":"cb016a04-d965-4ee7-b802-48c35f8c4b61","Type":"ContainerDied","Data":"631ad04b3020e46925cae0dd756ded7c060ca34a2796871e1d30dbd75901238d"}
Oct 29 23:07:14.500875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.500727    5428 scope.go:122] "RemoveContainer" containerID="98f1b02998d9a24acb564d647d8a9e7403693d102e113562dad09020bb765cfc"
Oct 29 23:07:14.514703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.514678    5428 scope.go:122] "RemoveContainer" containerID="98f1b02998d9a24acb564d647d8a9e7403693d102e113562dad09020bb765cfc"
Oct 29 23:07:14.515133 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:07:14.515082    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98f1b02998d9a24acb564d647d8a9e7403693d102e113562dad09020bb765cfc\": container with ID starting with 98f1b02998d9a24acb564d647d8a9e7403693d102e113562dad09020bb765cfc not found: ID does not exist" containerID="98f1b02998d9a24acb564d647d8a9e7403693d102e113562dad09020bb765cfc"
Oct 29 23:07:14.515259 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.515144    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f1b02998d9a24acb564d647d8a9e7403693d102e113562dad09020bb765cfc"} err="failed to get container status \"98f1b02998d9a24acb564d647d8a9e7403693d102e113562dad09020bb765cfc\": rpc error: code = NotFound desc = could not find container \"98f1b02998d9a24acb564d647d8a9e7403693d102e113562dad09020bb765cfc\": container with ID starting with 98f1b02998d9a24acb564d647d8a9e7403693d102e113562dad09020bb765cfc not found: ID does not exist"
Oct 29 23:07:14.551230 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.551198    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:14.551230 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.551233    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:14.551473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.551249    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhxbk\" (UniqueName: \"kubernetes.io/projected/cb016a04-d965-4ee7-b802-48c35f8c4b61-kube-api-access-vhxbk\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:14.551473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.551262    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb016a04-d965-4ee7-b802-48c35f8c4b61-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:14.557192 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.557168    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"]
Oct 29 23:07:14.561653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.561634    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-44a4e4-openshift-default-7b478dbcc5-8mtmp"]
Oct 29 23:07:14.781070 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.780989    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb016a04-d965-4ee7-b802-48c35f8c4b61" path="/var/lib/kubelet/pods/cb016a04-d965-4ee7-b802-48c35f8c4b61/volumes"
Oct 29 23:07:14.783028 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:14.783000    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:15.894925 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:15.894890    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:17.368906 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:07:17.368867    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:07:17.915439 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:17.915407    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:19.697514 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:19.697473    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:21.882104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:21.882072    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:23.564641 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:23.564600    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:25.152573 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.152532    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"]
Oct 29 23:07:25.154367 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.154335    5428 cpu_manager.go:414] "containerMap: removing container" podUID="cb016a04-d965-4ee7-b802-48c35f8c4b61" containerName="istio-proxy"
Oct 29 23:07:25.154367 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.154362    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb016a04-d965-4ee7-b802-48c35f8c4b61" containerName="istio-proxy"
Oct 29 23:07:25.154567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.154410    5428 cpu_manager.go:414] "containerMap: removing container" podUID="0748e57c-035e-47c3-a76f-bebe19de3139" containerName="istio-proxy"
Oct 29 23:07:25.154567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.154418    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0748e57c-035e-47c3-a76f-bebe19de3139" containerName="istio-proxy"
Oct 29 23:07:25.154567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.154446    5428 cpu_manager.go:414] "containerMap: removing container" podUID="c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2" containerName="payload-processing"
Oct 29 23:07:25.154567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.154455    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2" containerName="payload-processing"
Oct 29 23:07:25.154567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.154482    5428 cpu_manager.go:414] "containerMap: removing container" podUID="7242d814-03a8-4079-9202-ede3445634b3" containerName="maas-api"
Oct 29 23:07:25.154567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.154489    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7242d814-03a8-4079-9202-ede3445634b3" containerName="maas-api"
Oct 29 23:07:25.154567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.154520    5428 cpu_manager.go:414] "containerMap: removing container" podUID="be26e3f8-a847-49bc-aa52-c37190c3c2c6" containerName="payload-pre-processing"
Oct 29 23:07:25.154567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.154528    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be26e3f8-a847-49bc-aa52-c37190c3c2c6" containerName="payload-pre-processing"
Oct 29 23:07:25.154970 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.154859    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="0748e57c-035e-47c3-a76f-bebe19de3139" containerName="istio-proxy"
Oct 29 23:07:25.154970 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.154888    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6c5fcb5-e3e6-4bcb-9453-6a7c9868b2e2" containerName="payload-processing"
Oct 29 23:07:25.154970 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.154900    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="7242d814-03a8-4079-9202-ede3445634b3" containerName="maas-api"
Oct 29 23:07:25.154970 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.154921    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="be26e3f8-a847-49bc-aa52-c37190c3c2c6" containerName="payload-pre-processing"
Oct 29 23:07:25.154970 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.154948    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb016a04-d965-4ee7-b802-48c35f8c4b61" containerName="istio-proxy"
Oct 29 23:07:25.162431 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.162400    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:07:25.175794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.175768    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"]
Oct 29 23:07:25.182966 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.182934    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-cd000a-serving-cert\""
Oct 29 23:07:25.206397 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.206366    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq\" (UID: \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:07:25.206547 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.206521    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-metrics-certs\") pod \"maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq\" (UID: \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:07:25.206703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.206683    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq\" (UID: \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:07:25.206887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.206868    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqcdk\" (UniqueName: \"kubernetes.io/projected/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-kube-api-access-qqcdk\") pod \"maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq\" (UID: \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:07:25.266723 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.266686    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:25.307906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.307858    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq\" (UID: \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:07:25.308204 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:07:25.307981    5428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-serving-cert: secret "maas-api-e2e-praxis-b-cd000a-serving-cert" not found
Oct 29 23:07:25.308295 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:07:25.308281    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-maas-api-tls podName:1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5 nodeName:}" failed. No retries permitted until 2026-10-29 23:07:25.808255607 +0000 UTC m=+1449.151204336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-maas-api-tls") pod "maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq" (UID: "1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5") : secret "maas-api-e2e-praxis-b-cd000a-serving-cert" not found
Oct 29 23:07:25.308411 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.308392    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-metrics-certs\") pod \"maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq\" (UID: \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:07:25.308657 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.308634    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq\" (UID: \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:07:25.308974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.308951    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqcdk\" (UniqueName: \"kubernetes.io/projected/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-kube-api-access-qqcdk\") pod \"maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq\" (UID: \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:07:25.309735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.309711    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq\" (UID: \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:07:25.312283 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.312261    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-metrics-certs\") pod \"maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq\" (UID: \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:07:25.334908 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.334877    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqcdk\" (UniqueName: \"kubernetes.io/projected/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-kube-api-access-qqcdk\") pod \"maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq\" (UID: \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:07:25.384996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.384972    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:25.498217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.498126    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:25.818645 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.818558    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq\" (UID: \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:07:25.821027 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:25.821004    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq\" (UID: \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:07:26.079887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:26.079719    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:07:26.152054 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:26.152031    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:26.255392 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:26.255358    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn"]
Oct 29 23:07:26.288422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:26.288387    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b"]
Oct 29 23:07:26.397834 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:26.397151    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"]
Oct 29 23:07:26.410843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:26.410787    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:26.529357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:26.529323    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:26.603025 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:26.602984    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq" event={"ID":"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5","Type":"ContainerStarted","Data":"2323d392861da1fa18666e429eae49afa396f7715b93f1e4bf5d5517f370702b"}
Oct 29 23:07:26.659297 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:26.659202    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:27.356415 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.356304    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:27.557621 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.557586    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"]
Oct 29 23:07:27.562283 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.562259    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:07:27.591359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.591330    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-c23236-serving-cert\""
Oct 29 23:07:27.597060 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.597033    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"]
Oct 29 23:07:27.616285 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.616249    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq" event={"ID":"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5","Type":"ContainerStarted","Data":"2ad2ba0343adeadc910f33b97dfdbeae649b36144fb1e37c12ee6fe0af494035"}
Oct 29 23:07:27.616466 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.616316    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:07:27.647732 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.647703    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e77d6b76-753a-46f4-aab0-c35757e5f4bc-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c23236-994b96866-4kt28\" (UID: \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:07:27.647901 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.647749    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49vsq\" (UniqueName: \"kubernetes.io/projected/e77d6b76-753a-46f4-aab0-c35757e5f4bc-kube-api-access-49vsq\") pod \"maas-api-e2e-praxis-b-c23236-994b96866-4kt28\" (UID: \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:07:27.647901 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.647775    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e77d6b76-753a-46f4-aab0-c35757e5f4bc-metrics-certs\") pod \"maas-api-e2e-praxis-b-c23236-994b96866-4kt28\" (UID: \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:07:27.648322 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.648110    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77d6b76-753a-46f4-aab0-c35757e5f4bc-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c23236-994b96866-4kt28\" (UID: \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:07:27.666448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.666357    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq" podStartSLOduration=2.192081472 podStartE2EDuration="2.666339805s" totalImagesPullingTime="474.258333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:27.663726761 +0000 UTC m=+1451.006675507" watchObservedRunningTime="2026-10-29 23:07:27.666339805 +0000 UTC m=+1451.009288550"
Oct 29 23:07:27.749236 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.749204    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e77d6b76-753a-46f4-aab0-c35757e5f4bc-metrics-certs\") pod \"maas-api-e2e-praxis-b-c23236-994b96866-4kt28\" (UID: \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:07:27.749399 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.749350    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77d6b76-753a-46f4-aab0-c35757e5f4bc-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c23236-994b96866-4kt28\" (UID: \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:07:27.749469 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.749438    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e77d6b76-753a-46f4-aab0-c35757e5f4bc-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c23236-994b96866-4kt28\" (UID: \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:07:27.749531 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.749479    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49vsq\" (UniqueName: \"kubernetes.io/projected/e77d6b76-753a-46f4-aab0-c35757e5f4bc-kube-api-access-49vsq\") pod \"maas-api-e2e-praxis-b-c23236-994b96866-4kt28\" (UID: \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:07:27.749582 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:07:27.749539    5428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-serving-cert: secret "maas-api-e2e-praxis-b-c23236-serving-cert" not found
Oct 29 23:07:27.749632 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:07:27.749619    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e77d6b76-753a-46f4-aab0-c35757e5f4bc-maas-api-tls podName:e77d6b76-753a-46f4-aab0-c35757e5f4bc nodeName:}" failed. No retries permitted until 2026-10-29 23:07:28.249597059 +0000 UTC m=+1451.592545800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e77d6b76-753a-46f4-aab0-c35757e5f4bc-maas-api-tls") pod "maas-api-e2e-praxis-b-c23236-994b96866-4kt28" (UID: "e77d6b76-753a-46f4-aab0-c35757e5f4bc") : secret "maas-api-e2e-praxis-b-c23236-serving-cert" not found
Oct 29 23:07:27.749996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.749975    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77d6b76-753a-46f4-aab0-c35757e5f4bc-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c23236-994b96866-4kt28\" (UID: \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:07:27.751949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.751927    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e77d6b76-753a-46f4-aab0-c35757e5f4bc-metrics-certs\") pod \"maas-api-e2e-praxis-b-c23236-994b96866-4kt28\" (UID: \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:07:27.801247 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.801217    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49vsq\" (UniqueName: \"kubernetes.io/projected/e77d6b76-753a-46f4-aab0-c35757e5f4bc-kube-api-access-49vsq\") pod \"maas-api-e2e-praxis-b-c23236-994b96866-4kt28\" (UID: \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:07:27.812711 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.812681    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:27.944077 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:27.943999    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:28.257211 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:28.257117    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e77d6b76-753a-46f4-aab0-c35757e5f4bc-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c23236-994b96866-4kt28\" (UID: \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:07:28.259773 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:28.259750    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e77d6b76-753a-46f4-aab0-c35757e5f4bc-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c23236-994b96866-4kt28\" (UID: \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:07:28.477863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:28.477824    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:07:28.820108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:28.820080    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"]
Oct 29 23:07:28.820918 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:07:28.820892    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77d6b76_753a_46f4_aab0_c35757e5f4bc.slice/crio-0cb71508926f7e787211e6af65d6139029fe2d21031c0c6f418347393469c6f0 WatchSource:0}: Error finding container 0cb71508926f7e787211e6af65d6139029fe2d21031c0c6f418347393469c6f0: Status 404 returned error can't find the container with id 0cb71508926f7e787211e6af65d6139029fe2d21031c0c6f418347393469c6f0
Oct 29 23:07:29.346667 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:29.346628    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:29.518964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:29.518883    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:29.635274 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:29.635225    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28" event={"ID":"e77d6b76-753a-46f4-aab0-c35757e5f4bc","Type":"ContainerStarted","Data":"2ecb8222f2ceead08cba8c775d92a045df76dde12121f9757b513b7c2eac8d29"}
Oct 29 23:07:29.635469 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:29.635323    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28" event={"ID":"e77d6b76-753a-46f4-aab0-c35757e5f4bc","Type":"ContainerStarted","Data":"0cb71508926f7e787211e6af65d6139029fe2d21031c0c6f418347393469c6f0"}
Oct 29 23:07:29.635469 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:29.635358    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:07:29.668068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:29.668041    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:29.678758 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:29.678713    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28" podStartSLOduration=2.346555629 podStartE2EDuration="2.678699314s" totalImagesPullingTime="332.143685ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:29.677774709 +0000 UTC m=+1453.020723454" watchObservedRunningTime="2026-10-29 23:07:29.678699314 +0000 UTC m=+1453.021648082"
Oct 29 23:07:30.025682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:30.025652    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:30.241937 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:30.241902    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:30.363248 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:30.363171    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:30.850292 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:30.850248    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:31.256572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.256530    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn" podUID="847d1425-c094-4abc-b085-55d7c9c2bcbd" containerName="payload-processing" containerID="cri-o://52085e6c6e7a92ecb93e688a6fcffa4a0c43339dd1f9d16f97feb86862c6e936" gracePeriod=25
Oct 29 23:07:31.288820 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.288768    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b" podUID="d5294cfe-5196-4b3b-bfbe-60afe05d6cb5" containerName="payload-pre-processing" containerID="cri-o://379bcaa13b403ebdbe7f431fc918006e0e4e2ef19ca8cacdbcf45ea3c427c463" gracePeriod=25
Oct 29 23:07:31.572154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.572118    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn"
Oct 29 23:07:31.630381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.630351    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b"
Oct 29 23:07:31.630578 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.630508    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/847d1425-c094-4abc-b085-55d7c9c2bcbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/847d1425-c094-4abc-b085-55d7c9c2bcbd-plugins-config-volume\") pod \"847d1425-c094-4abc-b085-55d7c9c2bcbd\" (UID: \"847d1425-c094-4abc-b085-55d7c9c2bcbd\") "
Oct 29 23:07:31.630578 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.630562    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847d1425-c094-4abc-b085-55d7c9c2bcbd-kube-api-access-pcks9\" (UniqueName: \"kubernetes.io/projected/847d1425-c094-4abc-b085-55d7c9c2bcbd-kube-api-access-pcks9\") pod \"847d1425-c094-4abc-b085-55d7c9c2bcbd\" (UID: \"847d1425-c094-4abc-b085-55d7c9c2bcbd\") "
Oct 29 23:07:31.631059 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.631032    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/847d1425-c094-4abc-b085-55d7c9c2bcbd-plugins-config-volume" pod "847d1425-c094-4abc-b085-55d7c9c2bcbd" (UID: "847d1425-c094-4abc-b085-55d7c9c2bcbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:31.631436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.631415    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/847d1425-c094-4abc-b085-55d7c9c2bcbd-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:31.633158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.633129    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847d1425-c094-4abc-b085-55d7c9c2bcbd-kube-api-access-pcks9" pod "847d1425-c094-4abc-b085-55d7c9c2bcbd" (UID: "847d1425-c094-4abc-b085-55d7c9c2bcbd"). InnerVolumeSpecName "kube-api-access-pcks9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:31.651528 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.651501    5428 generic.go:356] "Generic (PLEG): container finished" podID="d5294cfe-5196-4b3b-bfbe-60afe05d6cb5" containerID="379bcaa13b403ebdbe7f431fc918006e0e4e2ef19ca8cacdbcf45ea3c427c463" exitCode=0
Oct 29 23:07:31.651636 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.651546    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b"
Oct 29 23:07:31.651636 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.651556    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b" event={"ID":"d5294cfe-5196-4b3b-bfbe-60afe05d6cb5","Type":"ContainerDied","Data":"379bcaa13b403ebdbe7f431fc918006e0e4e2ef19ca8cacdbcf45ea3c427c463"}
Oct 29 23:07:31.651636 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.651621    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b" event={"ID":"d5294cfe-5196-4b3b-bfbe-60afe05d6cb5","Type":"ContainerDied","Data":"62bc3b7e65a54afb0ded0f51f8591de29f1d1bb23ee309c86b4f2c4851ee8b57"}
Oct 29 23:07:31.651843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.651643    5428 scope.go:122] "RemoveContainer" containerID="379bcaa13b403ebdbe7f431fc918006e0e4e2ef19ca8cacdbcf45ea3c427c463"
Oct 29 23:07:31.653285 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.653260    5428 generic.go:356] "Generic (PLEG): container finished" podID="847d1425-c094-4abc-b085-55d7c9c2bcbd" containerID="52085e6c6e7a92ecb93e688a6fcffa4a0c43339dd1f9d16f97feb86862c6e936" exitCode=0
Oct 29 23:07:31.653416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.653315    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn"
Oct 29 23:07:31.653416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.653366    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn" event={"ID":"847d1425-c094-4abc-b085-55d7c9c2bcbd","Type":"ContainerDied","Data":"52085e6c6e7a92ecb93e688a6fcffa4a0c43339dd1f9d16f97feb86862c6e936"}
Oct 29 23:07:31.653416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.653406    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn" event={"ID":"847d1425-c094-4abc-b085-55d7c9c2bcbd","Type":"ContainerDied","Data":"01b633a8954c1a37aa49f453baa515e211eaaada1f607743fcc10be133db42f6"}
Oct 29 23:07:31.665314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.665292    5428 scope.go:122] "RemoveContainer" containerID="379bcaa13b403ebdbe7f431fc918006e0e4e2ef19ca8cacdbcf45ea3c427c463"
Oct 29 23:07:31.665622 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:07:31.665599    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379bcaa13b403ebdbe7f431fc918006e0e4e2ef19ca8cacdbcf45ea3c427c463\": container with ID starting with 379bcaa13b403ebdbe7f431fc918006e0e4e2ef19ca8cacdbcf45ea3c427c463 not found: ID does not exist" containerID="379bcaa13b403ebdbe7f431fc918006e0e4e2ef19ca8cacdbcf45ea3c427c463"
Oct 29 23:07:31.665717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.665631    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379bcaa13b403ebdbe7f431fc918006e0e4e2ef19ca8cacdbcf45ea3c427c463"} err="failed to get container status \"379bcaa13b403ebdbe7f431fc918006e0e4e2ef19ca8cacdbcf45ea3c427c463\": rpc error: code = NotFound desc = could not find container \"379bcaa13b403ebdbe7f431fc918006e0e4e2ef19ca8cacdbcf45ea3c427c463\": container with ID starting with 379bcaa13b403ebdbe7f431fc918006e0e4e2ef19ca8cacdbcf45ea3c427c463 not found: ID does not exist"
Oct 29 23:07:31.665717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.665653    5428 scope.go:122] "RemoveContainer" containerID="52085e6c6e7a92ecb93e688a6fcffa4a0c43339dd1f9d16f97feb86862c6e936"
Oct 29 23:07:31.703688 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.679241    5428 scope.go:122] "RemoveContainer" containerID="52085e6c6e7a92ecb93e688a6fcffa4a0c43339dd1f9d16f97feb86862c6e936"
Oct 29 23:07:31.703688 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:07:31.679586    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52085e6c6e7a92ecb93e688a6fcffa4a0c43339dd1f9d16f97feb86862c6e936\": container with ID starting with 52085e6c6e7a92ecb93e688a6fcffa4a0c43339dd1f9d16f97feb86862c6e936 not found: ID does not exist" containerID="52085e6c6e7a92ecb93e688a6fcffa4a0c43339dd1f9d16f97feb86862c6e936"
Oct 29 23:07:31.703688 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.679617    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52085e6c6e7a92ecb93e688a6fcffa4a0c43339dd1f9d16f97feb86862c6e936"} err="failed to get container status \"52085e6c6e7a92ecb93e688a6fcffa4a0c43339dd1f9d16f97feb86862c6e936\": rpc error: code = NotFound desc = could not find container \"52085e6c6e7a92ecb93e688a6fcffa4a0c43339dd1f9d16f97feb86862c6e936\": container with ID starting with 52085e6c6e7a92ecb93e688a6fcffa4a0c43339dd1f9d16f97feb86862c6e936 not found: ID does not exist"
Oct 29 23:07:31.732993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.732967    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5294cfe-5196-4b3b-bfbe-60afe05d6cb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5294cfe-5196-4b3b-bfbe-60afe05d6cb5-plugins-config-volume\") pod \"d5294cfe-5196-4b3b-bfbe-60afe05d6cb5\" (UID: \"d5294cfe-5196-4b3b-bfbe-60afe05d6cb5\") "
Oct 29 23:07:31.733121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.733010    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5294cfe-5196-4b3b-bfbe-60afe05d6cb5-kube-api-access-b6vr5\" (UniqueName: \"kubernetes.io/projected/d5294cfe-5196-4b3b-bfbe-60afe05d6cb5-kube-api-access-b6vr5\") pod \"d5294cfe-5196-4b3b-bfbe-60afe05d6cb5\" (UID: \"d5294cfe-5196-4b3b-bfbe-60afe05d6cb5\") "
Oct 29 23:07:31.733405 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.733381    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5294cfe-5196-4b3b-bfbe-60afe05d6cb5-plugins-config-volume" pod "d5294cfe-5196-4b3b-bfbe-60afe05d6cb5" (UID: "d5294cfe-5196-4b3b-bfbe-60afe05d6cb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:31.733545 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.733532    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5294cfe-5196-4b3b-bfbe-60afe05d6cb5-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:31.733579 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.733552    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcks9\" (UniqueName: \"kubernetes.io/projected/847d1425-c094-4abc-b085-55d7c9c2bcbd-kube-api-access-pcks9\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:31.735169 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.735149    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5294cfe-5196-4b3b-bfbe-60afe05d6cb5-kube-api-access-b6vr5" pod "d5294cfe-5196-4b3b-bfbe-60afe05d6cb5" (UID: "d5294cfe-5196-4b3b-bfbe-60afe05d6cb5"). InnerVolumeSpecName "kube-api-access-b6vr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:31.742333 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.742307    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn"]
Oct 29 23:07:31.749311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.749289    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-921680-6dc8bd7fd4-g7tfn"]
Oct 29 23:07:31.834610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.834542    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6vr5\" (UniqueName: \"kubernetes.io/projected/d5294cfe-5196-4b3b-bfbe-60afe05d6cb5-kube-api-access-b6vr5\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:31.999633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:31.999598    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b"]
Oct 29 23:07:32.003506 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:32.003484    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-921680-78ffbb946c-2d65b"]
Oct 29 23:07:32.305311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:32.305279    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:32.778819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:32.778776    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="847d1425-c094-4abc-b085-55d7c9c2bcbd" path="/var/lib/kubelet/pods/847d1425-c094-4abc-b085-55d7c9c2bcbd/volumes"
Oct 29 23:07:32.779293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:32.779278    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5294cfe-5196-4b3b-bfbe-60afe05d6cb5" path="/var/lib/kubelet/pods/d5294cfe-5196-4b3b-bfbe-60afe05d6cb5/volumes"
Oct 29 23:07:33.632051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:33.632016    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:07:34.253042 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:34.253007    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:35.648525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:35.648495    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:07:36.506037 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:36.505996    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:37.148538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:37.148500    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:37.698920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:37.698883    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:37.811551 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:37.811518    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:38.066743 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:38.066662    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:38.194342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:38.194305    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:38.990074 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:38.990040    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:39.139217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:39.139176    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:39.426257 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:39.426221    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:39.628049 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:39.628017    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:39.780526 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:39.780444    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:40.124226 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:40.124195    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:40.401873 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:40.401751    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:41.681249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:41.681216    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:43.481324 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:43.481290    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:44.465296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:44.465258    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:44.614745 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:44.614706    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:44.845721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:44.845647    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:45.406227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:45.406192    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"]
Oct 29 23:07:45.406429 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:45.406406    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr" podUID="53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" containerName="istio-proxy" containerID="cri-o://cc0384e2c90a2ac302230f7847b2f40f6d2dfb74e7d38c2f56219ade6454ce63" gracePeriod=30
Oct 29 23:07:45.523879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:45.523850    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:46.144561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:46.144530    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:46.267289 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:46.267259    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:47.049912 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:47.049878    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:47.191590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:47.191554    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:47.375161 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:47.375128    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"]
Oct 29 23:07:47.375403 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:47.375370    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg" podUID="d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" containerName="istio-proxy" containerID="cri-o://d8c8c7b639148250ac1e83d28a0902d41d54d65a1f6a9cf3293c0c37b9aeedd7" gracePeriod=30
Oct 29 23:07:47.388040 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:07:47.388006    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:07:47.410863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:47.410835    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:48.224713 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:48.224676    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:50.637326 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.637291    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:50.755985 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.755954    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 23:07:50.756955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.756923    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:50.833441 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.833409    5428 generic.go:356] "Generic (PLEG): container finished" podID="53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" containerID="cc0384e2c90a2ac302230f7847b2f40f6d2dfb74e7d38c2f56219ade6454ce63" exitCode=0
Oct 29 23:07:50.833659 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.833461    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr" event={"ID":"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28","Type":"ContainerDied","Data":"cc0384e2c90a2ac302230f7847b2f40f6d2dfb74e7d38c2f56219ade6454ce63"}
Oct 29 23:07:50.833659 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.833498    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr" event={"ID":"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28","Type":"ContainerDied","Data":"600f7461c71415b3afcd877ad02cd72ca3ed3bba90c585954edeffe489d276e3"}
Oct 29 23:07:50.833659 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.833522    5428 scope.go:122] "RemoveContainer" containerID="cc0384e2c90a2ac302230f7847b2f40f6d2dfb74e7d38c2f56219ade6454ce63"
Oct 29 23:07:50.833659 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.833647    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"
Oct 29 23:07:50.864298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.862017    5428 scope.go:122] "RemoveContainer" containerID="cc0384e2c90a2ac302230f7847b2f40f6d2dfb74e7d38c2f56219ade6454ce63"
Oct 29 23:07:50.864298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.863241    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istiod-ca-cert\") pod \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") "
Oct 29 23:07:50.864298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.863300    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-podinfo\") pod \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") "
Oct 29 23:07:50.864298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.863346    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-workload-socket\") pod \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") "
Oct 29 23:07:50.864298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.863379    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-envoy\") pod \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") "
Oct 29 23:07:50.864298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.863514    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-credential-socket\") pod \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") "
Oct 29 23:07:50.864298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.863552    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-token\" (UniqueName: \"kubernetes.io/projected/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-token\") pod \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") "
Oct 29 23:07:50.864298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.863643    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-kube-api-access-b6d84\" (UniqueName: \"kubernetes.io/projected/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-kube-api-access-b6d84\") pod \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") "
Oct 29 23:07:50.864298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.863783    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-workload-certs\") pod \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") "
Oct 29 23:07:50.864298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.863926    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-data\") pod \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\" (UID: \"53e3c3eb-c2d6-4c8d-b1db-c953a8964b28\") "
Oct 29 23:07:50.865642 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.865612    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-credential-socket" pod "53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" (UID: "53e3c3eb-c2d6-4c8d-b1db-c953a8964b28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:50.865940 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.865905    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-data" pod "53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" (UID: "53e3c3eb-c2d6-4c8d-b1db-c953a8964b28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:50.867229 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.866764    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-workload-certs" pod "53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" (UID: "53e3c3eb-c2d6-4c8d-b1db-c953a8964b28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:50.867229 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:07:50.866917    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc0384e2c90a2ac302230f7847b2f40f6d2dfb74e7d38c2f56219ade6454ce63\": container with ID starting with cc0384e2c90a2ac302230f7847b2f40f6d2dfb74e7d38c2f56219ade6454ce63 not found: ID does not exist" containerID="cc0384e2c90a2ac302230f7847b2f40f6d2dfb74e7d38c2f56219ade6454ce63"
Oct 29 23:07:50.867229 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.866948    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc0384e2c90a2ac302230f7847b2f40f6d2dfb74e7d38c2f56219ade6454ce63"} err="failed to get container status \"cc0384e2c90a2ac302230f7847b2f40f6d2dfb74e7d38c2f56219ade6454ce63\": rpc error: code = NotFound desc = could not find container \"cc0384e2c90a2ac302230f7847b2f40f6d2dfb74e7d38c2f56219ade6454ce63\": container with ID starting with cc0384e2c90a2ac302230f7847b2f40f6d2dfb74e7d38c2f56219ade6454ce63 not found: ID does not exist"
Oct 29 23:07:50.867783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.867452    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-workload-socket" pod "53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" (UID: "53e3c3eb-c2d6-4c8d-b1db-c953a8964b28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:50.867783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.867518    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-envoy" pod "53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" (UID: "53e3c3eb-c2d6-4c8d-b1db-c953a8964b28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:50.869265 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.869225    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istiod-ca-cert" pod "53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" (UID: "53e3c3eb-c2d6-4c8d-b1db-c953a8964b28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:50.869858 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.869791    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-kube-api-access-b6d84" pod "53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" (UID: "53e3c3eb-c2d6-4c8d-b1db-c953a8964b28"). InnerVolumeSpecName "kube-api-access-b6d84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:50.875846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.873380    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-token" pod "53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" (UID: "53e3c3eb-c2d6-4c8d-b1db-c953a8964b28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:50.877295 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.877266    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-podinfo" pod "53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" (UID: "53e3c3eb-c2d6-4c8d-b1db-c953a8964b28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:07:50.966166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.966124    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:50.966166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.966162    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:50.966357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.966177    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6d84\" (UniqueName: \"kubernetes.io/projected/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-kube-api-access-b6d84\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:50.966357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.966187    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:50.966357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.966195    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:50.966357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.966203    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:50.966357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.966210    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:50.966357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.966218    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:50.966357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:50.966229    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:51.185632 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:51.185592    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"]
Oct 29 23:07:51.196791 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:51.196752    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-zpxqr"]
Oct 29 23:07:51.339680 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:51.339645    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:52.684113 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.684087    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:07:52.688666 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.688644    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-token\" (UniqueName: \"kubernetes.io/projected/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-token\") pod \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") "
Oct 29 23:07:52.688773 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.688745    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-data\") pod \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") "
Oct 29 23:07:52.688840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.688776    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-podinfo\") pod \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") "
Oct 29 23:07:52.688911 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.688845    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-envoy\") pod \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") "
Oct 29 23:07:52.688911 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.688886    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-credential-socket\") pod \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") "
Oct 29 23:07:52.689020 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.688959    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istiod-ca-cert\") pod \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") "
Oct 29 23:07:52.689073 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.689018    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-workload-certs\") pod \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") "
Oct 29 23:07:52.689136 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.689074    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-data" pod "d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" (UID: "d85de9d4-72c8-4340-ae0d-f9c8735f5a4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:52.689193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.689135    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-kube-api-access-pvdlq\" (UniqueName: \"kubernetes.io/projected/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-kube-api-access-pvdlq\") pod \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") "
Oct 29 23:07:52.689250 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.689228    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-credential-socket" pod "d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" (UID: "d85de9d4-72c8-4340-ae0d-f9c8735f5a4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:52.689302 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.689242    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-workload-certs" pod "d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" (UID: "d85de9d4-72c8-4340-ae0d-f9c8735f5a4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:52.689358 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.689326    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-workload-socket\") pod \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\" (UID: \"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f\") "
Oct 29 23:07:52.689439 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.689416    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istiod-ca-cert" pod "d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" (UID: "d85de9d4-72c8-4340-ae0d-f9c8735f5a4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:52.689740 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.689715    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-workload-socket" pod "d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" (UID: "d85de9d4-72c8-4340-ae0d-f9c8735f5a4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:52.689985 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.689967    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:52.690076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.689988    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:52.690076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.689999    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:52.690076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.690008    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:52.690076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.690017    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:52.690971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.690937    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-podinfo" pod "d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" (UID: "d85de9d4-72c8-4340-ae0d-f9c8735f5a4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:07:52.691270 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.691247    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-envoy" pod "d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" (UID: "d85de9d4-72c8-4340-ae0d-f9c8735f5a4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:07:52.691270 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.691260    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-token" pod "d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" (UID: "d85de9d4-72c8-4340-ae0d-f9c8735f5a4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:52.691381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.691360    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-kube-api-access-pvdlq" pod "d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" (UID: "d85de9d4-72c8-4340-ae0d-f9c8735f5a4f"). InnerVolumeSpecName "kube-api-access-pvdlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:52.780940 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.780906    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" path="/var/lib/kubelet/pods/53e3c3eb-c2d6-4c8d-b1db-c953a8964b28/volumes"
Oct 29 23:07:52.791410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.791381    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvdlq\" (UniqueName: \"kubernetes.io/projected/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-kube-api-access-pvdlq\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:52.791511 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.791419    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:52.791511 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.791434    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:52.791511 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.791447    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:07:52.852897 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.852860    5428 generic.go:356] "Generic (PLEG): container finished" podID="d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" containerID="d8c8c7b639148250ac1e83d28a0902d41d54d65a1f6a9cf3293c0c37b9aeedd7" exitCode=0
Oct 29 23:07:52.853050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.852938    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg" event={"ID":"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f","Type":"ContainerDied","Data":"d8c8c7b639148250ac1e83d28a0902d41d54d65a1f6a9cf3293c0c37b9aeedd7"}
Oct 29 23:07:52.853050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.852953    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"
Oct 29 23:07:52.853050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.852992    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg" event={"ID":"d85de9d4-72c8-4340-ae0d-f9c8735f5a4f","Type":"ContainerDied","Data":"d0fedd6e166c452a3345526df81925bafdf4fd0274f32a174aaa13421f71c197"}
Oct 29 23:07:52.853050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.853015    5428 scope.go:122] "RemoveContainer" containerID="d8c8c7b639148250ac1e83d28a0902d41d54d65a1f6a9cf3293c0c37b9aeedd7"
Oct 29 23:07:52.864730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.864700    5428 scope.go:122] "RemoveContainer" containerID="d8c8c7b639148250ac1e83d28a0902d41d54d65a1f6a9cf3293c0c37b9aeedd7"
Oct 29 23:07:52.865075 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:07:52.865046    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c8c7b639148250ac1e83d28a0902d41d54d65a1f6a9cf3293c0c37b9aeedd7\": container with ID starting with d8c8c7b639148250ac1e83d28a0902d41d54d65a1f6a9cf3293c0c37b9aeedd7 not found: ID does not exist" containerID="d8c8c7b639148250ac1e83d28a0902d41d54d65a1f6a9cf3293c0c37b9aeedd7"
Oct 29 23:07:52.865174 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.865083    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c8c7b639148250ac1e83d28a0902d41d54d65a1f6a9cf3293c0c37b9aeedd7"} err="failed to get container status \"d8c8c7b639148250ac1e83d28a0902d41d54d65a1f6a9cf3293c0c37b9aeedd7\": rpc error: code = NotFound desc = could not find container \"d8c8c7b639148250ac1e83d28a0902d41d54d65a1f6a9cf3293c0c37b9aeedd7\": container with ID starting with d8c8c7b639148250ac1e83d28a0902d41d54d65a1f6a9cf3293c0c37b9aeedd7 not found: ID does not exist"
Oct 29 23:07:52.910704 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.910634    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"]
Oct 29 23:07:52.914335 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:52.914314    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b0c72c-openshift-default-858fc44f5-m2hqg"]
Oct 29 23:07:53.368724 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:53.368694    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:53.496692 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:53.496658    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:54.779661 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:54.779627    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" path="/var/lib/kubelet/pods/d85de9d4-72c8-4340-ae0d-f9c8735f5a4f/volumes"
Oct 29 23:07:55.498737 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:55.498695    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:57.422865 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:57.422829    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:58.581684 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:58.581646    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg"]
Oct 29 23:07:58.614093 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:58.614053    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56"]
Oct 29 23:07:58.734651 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:58.734615    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:58.858196 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:58.858084    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:59.336477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:07:59.336442    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:01.371854 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.371814    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"]
Oct 29 23:08:01.372429 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.372074    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls" podUID="cb5221e1-3731-4150-aaba-89507c4b0cc8" containerName="maas-api" containerID="cri-o://67fe58e8ecb908ff49c81e554a6e7d7f667239ab7edcea1abda8834cfd4a82a8" gracePeriod=30
Oct 29 23:08:01.446264 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.446232    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:01.698598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.698566    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:08:01.717645 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.717612    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb5221e1-3731-4150-aaba-89507c4b0cc8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5221e1-3731-4150-aaba-89507c4b0cc8-openshift-service-ca\") pod \"cb5221e1-3731-4150-aaba-89507c4b0cc8\" (UID: \"cb5221e1-3731-4150-aaba-89507c4b0cc8\") "
Oct 29 23:08:01.717795 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.717780    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb5221e1-3731-4150-aaba-89507c4b0cc8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb5221e1-3731-4150-aaba-89507c4b0cc8-maas-api-tls\") pod \"cb5221e1-3731-4150-aaba-89507c4b0cc8\" (UID: \"cb5221e1-3731-4150-aaba-89507c4b0cc8\") "
Oct 29 23:08:01.717938 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.717914    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb5221e1-3731-4150-aaba-89507c4b0cc8-kube-api-access-bdzzh\" (UniqueName: \"kubernetes.io/projected/cb5221e1-3731-4150-aaba-89507c4b0cc8-kube-api-access-bdzzh\") pod \"cb5221e1-3731-4150-aaba-89507c4b0cc8\" (UID: \"cb5221e1-3731-4150-aaba-89507c4b0cc8\") "
Oct 29 23:08:01.718100 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.718072    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5221e1-3731-4150-aaba-89507c4b0cc8-openshift-service-ca" pod "cb5221e1-3731-4150-aaba-89507c4b0cc8" (UID: "cb5221e1-3731-4150-aaba-89507c4b0cc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:01.718238 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.718122    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb5221e1-3731-4150-aaba-89507c4b0cc8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb5221e1-3731-4150-aaba-89507c4b0cc8-metrics-certs\") pod \"cb5221e1-3731-4150-aaba-89507c4b0cc8\" (UID: \"cb5221e1-3731-4150-aaba-89507c4b0cc8\") "
Oct 29 23:08:01.718705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.718685    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5221e1-3731-4150-aaba-89507c4b0cc8-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:01.720840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.720372    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb5221e1-3731-4150-aaba-89507c4b0cc8-metrics-certs" pod "cb5221e1-3731-4150-aaba-89507c4b0cc8" (UID: "cb5221e1-3731-4150-aaba-89507c4b0cc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:01.720840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.720455    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb5221e1-3731-4150-aaba-89507c4b0cc8-maas-api-tls" pod "cb5221e1-3731-4150-aaba-89507c4b0cc8" (UID: "cb5221e1-3731-4150-aaba-89507c4b0cc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:01.720840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.720479    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5221e1-3731-4150-aaba-89507c4b0cc8-kube-api-access-bdzzh" pod "cb5221e1-3731-4150-aaba-89507c4b0cc8" (UID: "cb5221e1-3731-4150-aaba-89507c4b0cc8"). InnerVolumeSpecName "kube-api-access-bdzzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:01.820427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.820397    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb5221e1-3731-4150-aaba-89507c4b0cc8-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:01.820427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.820423    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb5221e1-3731-4150-aaba-89507c4b0cc8-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:01.820427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.820433    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdzzh\" (UniqueName: \"kubernetes.io/projected/cb5221e1-3731-4150-aaba-89507c4b0cc8-kube-api-access-bdzzh\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:01.930111 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.930003    5428 generic.go:356] "Generic (PLEG): container finished" podID="cb5221e1-3731-4150-aaba-89507c4b0cc8" containerID="67fe58e8ecb908ff49c81e554a6e7d7f667239ab7edcea1abda8834cfd4a82a8" exitCode=0
Oct 29 23:08:01.930111 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.930043    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls" event={"ID":"cb5221e1-3731-4150-aaba-89507c4b0cc8","Type":"ContainerDied","Data":"67fe58e8ecb908ff49c81e554a6e7d7f667239ab7edcea1abda8834cfd4a82a8"}
Oct 29 23:08:01.930111 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.930063    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"
Oct 29 23:08:01.930397 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.930118    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls" event={"ID":"cb5221e1-3731-4150-aaba-89507c4b0cc8","Type":"ContainerDied","Data":"1f59d49fbe90ba6e3bb69f84cc42972b5b81699b836c3ce13da1db64c2aa8a55"}
Oct 29 23:08:01.930397 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.930158    5428 scope.go:122] "RemoveContainer" containerID="67fe58e8ecb908ff49c81e554a6e7d7f667239ab7edcea1abda8834cfd4a82a8"
Oct 29 23:08:01.945191 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.944996    5428 scope.go:122] "RemoveContainer" containerID="67fe58e8ecb908ff49c81e554a6e7d7f667239ab7edcea1abda8834cfd4a82a8"
Oct 29 23:08:01.945519 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:08:01.945476    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67fe58e8ecb908ff49c81e554a6e7d7f667239ab7edcea1abda8834cfd4a82a8\": container with ID starting with 67fe58e8ecb908ff49c81e554a6e7d7f667239ab7edcea1abda8834cfd4a82a8 not found: ID does not exist" containerID="67fe58e8ecb908ff49c81e554a6e7d7f667239ab7edcea1abda8834cfd4a82a8"
Oct 29 23:08:01.945630 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.945516    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67fe58e8ecb908ff49c81e554a6e7d7f667239ab7edcea1abda8834cfd4a82a8"} err="failed to get container status \"67fe58e8ecb908ff49c81e554a6e7d7f667239ab7edcea1abda8834cfd4a82a8\": rpc error: code = NotFound desc = could not find container \"67fe58e8ecb908ff49c81e554a6e7d7f667239ab7edcea1abda8834cfd4a82a8\": container with ID starting with 67fe58e8ecb908ff49c81e554a6e7d7f667239ab7edcea1abda8834cfd4a82a8 not found: ID does not exist"
Oct 29 23:08:01.986628 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.986591    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"]
Oct 29 23:08:01.992591 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:01.992550    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0941c8-7db4f97c9f-gn2ls"]
Oct 29 23:08:02.061550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:02.061456    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:02.344365 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:02.344287    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:02.675682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:02.675647    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:02.784341 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:02.784311    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb5221e1-3731-4150-aaba-89507c4b0cc8" path="/var/lib/kubelet/pods/cb5221e1-3731-4150-aaba-89507c4b0cc8/volumes"
Oct 29 23:08:02.896604 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:02.896573    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:03.582315 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.582274    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg" podUID="82514398-fdc2-46df-b04f-9d5429af45e8" containerName="payload-processing" containerID="cri-o://3b5433942519c7eb1ff119775c0c4043603841f713af972aea5367f803abe773" gracePeriod=25
Oct 29 23:08:03.615095 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.615066    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56" podUID="c946b973-a30a-432c-8ee0-5ea240c5ee26" containerName="payload-pre-processing" containerID="cri-o://78f05a9b7514adec473becb99bb2f44c22e0f8ff1235160dfc72356ca1bb12f1" gracePeriod=25
Oct 29 23:08:03.913739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.913712    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg"
Oct 29 23:08:03.947668 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.947632    5428 generic.go:356] "Generic (PLEG): container finished" podID="82514398-fdc2-46df-b04f-9d5429af45e8" containerID="3b5433942519c7eb1ff119775c0c4043603841f713af972aea5367f803abe773" exitCode=0
Oct 29 23:08:03.947823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.947678    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg"
Oct 29 23:08:03.947823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.947714    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg" event={"ID":"82514398-fdc2-46df-b04f-9d5429af45e8","Type":"ContainerDied","Data":"3b5433942519c7eb1ff119775c0c4043603841f713af972aea5367f803abe773"}
Oct 29 23:08:03.947823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.947767    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg" event={"ID":"82514398-fdc2-46df-b04f-9d5429af45e8","Type":"ContainerDied","Data":"298ca4c7081e0fe2accf18b55a0350024c9013726f219f0ced5ed4065b546160"}
Oct 29 23:08:03.947823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.947793    5428 scope.go:122] "RemoveContainer" containerID="3b5433942519c7eb1ff119775c0c4043603841f713af972aea5367f803abe773"
Oct 29 23:08:03.948856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.948731    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56"
Oct 29 23:08:03.949512 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.949488    5428 generic.go:356] "Generic (PLEG): container finished" podID="c946b973-a30a-432c-8ee0-5ea240c5ee26" containerID="78f05a9b7514adec473becb99bb2f44c22e0f8ff1235160dfc72356ca1bb12f1" exitCode=0
Oct 29 23:08:03.949613 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.949558    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56" event={"ID":"c946b973-a30a-432c-8ee0-5ea240c5ee26","Type":"ContainerDied","Data":"78f05a9b7514adec473becb99bb2f44c22e0f8ff1235160dfc72356ca1bb12f1"}
Oct 29 23:08:03.949613 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.949602    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56" event={"ID":"c946b973-a30a-432c-8ee0-5ea240c5ee26","Type":"ContainerDied","Data":"912303f6693ceb6499e72ecee858049403645f17fbec4f5f736f30a8c4108867"}
Oct 29 23:08:03.950877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.950853    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82514398-fdc2-46df-b04f-9d5429af45e8-kube-api-access-cbghr\" (UniqueName: \"kubernetes.io/projected/82514398-fdc2-46df-b04f-9d5429af45e8-kube-api-access-cbghr\") pod \"82514398-fdc2-46df-b04f-9d5429af45e8\" (UID: \"82514398-fdc2-46df-b04f-9d5429af45e8\") "
Oct 29 23:08:03.951105 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.951088    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82514398-fdc2-46df-b04f-9d5429af45e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82514398-fdc2-46df-b04f-9d5429af45e8-plugins-config-volume\") pod \"82514398-fdc2-46df-b04f-9d5429af45e8\" (UID: \"82514398-fdc2-46df-b04f-9d5429af45e8\") "
Oct 29 23:08:03.951692 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.951665    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82514398-fdc2-46df-b04f-9d5429af45e8-plugins-config-volume" pod "82514398-fdc2-46df-b04f-9d5429af45e8" (UID: "82514398-fdc2-46df-b04f-9d5429af45e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:03.953098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.953073    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82514398-fdc2-46df-b04f-9d5429af45e8-kube-api-access-cbghr" pod "82514398-fdc2-46df-b04f-9d5429af45e8" (UID: "82514398-fdc2-46df-b04f-9d5429af45e8"). InnerVolumeSpecName "kube-api-access-cbghr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:03.964690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.964279    5428 scope.go:122] "RemoveContainer" containerID="3b5433942519c7eb1ff119775c0c4043603841f713af972aea5367f803abe773"
Oct 29 23:08:03.964759 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:08:03.964703    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5433942519c7eb1ff119775c0c4043603841f713af972aea5367f803abe773\": container with ID starting with 3b5433942519c7eb1ff119775c0c4043603841f713af972aea5367f803abe773 not found: ID does not exist" containerID="3b5433942519c7eb1ff119775c0c4043603841f713af972aea5367f803abe773"
Oct 29 23:08:03.964759 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.964736    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5433942519c7eb1ff119775c0c4043603841f713af972aea5367f803abe773"} err="failed to get container status \"3b5433942519c7eb1ff119775c0c4043603841f713af972aea5367f803abe773\": rpc error: code = NotFound desc = could not find container \"3b5433942519c7eb1ff119775c0c4043603841f713af972aea5367f803abe773\": container with ID starting with 3b5433942519c7eb1ff119775c0c4043603841f713af972aea5367f803abe773 not found: ID does not exist"
Oct 29 23:08:03.965007 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.964758    5428 scope.go:122] "RemoveContainer" containerID="78f05a9b7514adec473becb99bb2f44c22e0f8ff1235160dfc72356ca1bb12f1"
Oct 29 23:08:03.979533 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.979511    5428 scope.go:122] "RemoveContainer" containerID="78f05a9b7514adec473becb99bb2f44c22e0f8ff1235160dfc72356ca1bb12f1"
Oct 29 23:08:03.980033 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:08:03.980006    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f05a9b7514adec473becb99bb2f44c22e0f8ff1235160dfc72356ca1bb12f1\": container with ID starting with 78f05a9b7514adec473becb99bb2f44c22e0f8ff1235160dfc72356ca1bb12f1 not found: ID does not exist" containerID="78f05a9b7514adec473becb99bb2f44c22e0f8ff1235160dfc72356ca1bb12f1"
Oct 29 23:08:03.980126 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:03.980042    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f05a9b7514adec473becb99bb2f44c22e0f8ff1235160dfc72356ca1bb12f1"} err="failed to get container status \"78f05a9b7514adec473becb99bb2f44c22e0f8ff1235160dfc72356ca1bb12f1\": rpc error: code = NotFound desc = could not find container \"78f05a9b7514adec473becb99bb2f44c22e0f8ff1235160dfc72356ca1bb12f1\": container with ID starting with 78f05a9b7514adec473becb99bb2f44c22e0f8ff1235160dfc72356ca1bb12f1 not found: ID does not exist"
Oct 29 23:08:04.049397 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:04.049369    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:04.052563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:04.052533    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c946b973-a30a-432c-8ee0-5ea240c5ee26-kube-api-access-jljbt\" (UniqueName: \"kubernetes.io/projected/c946b973-a30a-432c-8ee0-5ea240c5ee26-kube-api-access-jljbt\") pod \"c946b973-a30a-432c-8ee0-5ea240c5ee26\" (UID: \"c946b973-a30a-432c-8ee0-5ea240c5ee26\") "
Oct 29 23:08:04.052936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:04.052920    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c946b973-a30a-432c-8ee0-5ea240c5ee26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c946b973-a30a-432c-8ee0-5ea240c5ee26-plugins-config-volume\") pod \"c946b973-a30a-432c-8ee0-5ea240c5ee26\" (UID: \"c946b973-a30a-432c-8ee0-5ea240c5ee26\") "
Oct 29 23:08:04.053328 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:04.053310    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbghr\" (UniqueName: \"kubernetes.io/projected/82514398-fdc2-46df-b04f-9d5429af45e8-kube-api-access-cbghr\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:04.053386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:04.053331    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82514398-fdc2-46df-b04f-9d5429af45e8-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:04.053386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:04.053320    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c946b973-a30a-432c-8ee0-5ea240c5ee26-plugins-config-volume" pod "c946b973-a30a-432c-8ee0-5ea240c5ee26" (UID: "c946b973-a30a-432c-8ee0-5ea240c5ee26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:04.054615 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:04.054598    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c946b973-a30a-432c-8ee0-5ea240c5ee26-kube-api-access-jljbt" pod "c946b973-a30a-432c-8ee0-5ea240c5ee26" (UID: "c946b973-a30a-432c-8ee0-5ea240c5ee26"). InnerVolumeSpecName "kube-api-access-jljbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:04.158959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:04.157991    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c946b973-a30a-432c-8ee0-5ea240c5ee26-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:04.158959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:04.158024    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jljbt\" (UniqueName: \"kubernetes.io/projected/c946b973-a30a-432c-8ee0-5ea240c5ee26-kube-api-access-jljbt\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:04.226780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:04.226744    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:04.297290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:04.297243    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg"]
Oct 29 23:08:04.301015 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:04.300987    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57b28e-8685dbb9bb-s96lg"]
Oct 29 23:08:04.780631 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:04.780595    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82514398-fdc2-46df-b04f-9d5429af45e8" path="/var/lib/kubelet/pods/82514398-fdc2-46df-b04f-9d5429af45e8/volumes"
Oct 29 23:08:04.958996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:04.958962    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56"
Oct 29 23:08:05.012465 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:05.012430    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56"]
Oct 29 23:08:05.041307 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:05.041270    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57b28e-7c899f5db7-t7n56"]
Oct 29 23:08:05.611919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:05.611881    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:06.785978 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:06.785769    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c946b973-a30a-432c-8ee0-5ea240c5ee26" path="/var/lib/kubelet/pods/c946b973-a30a-432c-8ee0-5ea240c5ee26/volumes"
Oct 29 23:08:08.126619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:08.126583    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:08.873659 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:08.873626    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:09.190792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:09.190717    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:09.303515 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:09.303473    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:09.556617 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:09.556539    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:10.936370 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:10.936334    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:13.002628 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:13.002597    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:14.818233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:14.818201    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:15.292057 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:15.292024    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:17.088362 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:17.088325    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:17.368022 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:08:17.367979    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:08:18.861765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:18.861729    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:20.534497 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:20.534464    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:20.557302 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:20.557270    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2"]
Oct 29 23:08:20.699071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:20.699032    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:20.819867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:20.819732    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:21.271646 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:21.271596    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2"]
Oct 29 23:08:21.413911 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:21.413849    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:21.557636 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:21.557547    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:22.539897 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:22.539866    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:22.652518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:22.652486    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:23.855711 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:23.855672    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:24.354959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:24.354919    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:24.653526 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:24.653496    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:24.967084 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:24.966986    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:25.558257 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:25.558195    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2" podUID="92ea570b-9973-4fd2-b18a-aa32192ba3d8" containerName="payload-processing" containerID="cri-o://9135a195457e41d388b1a228ab52fbad7c2dfe7f82a52adc1178c20a17c42acf" gracePeriod=25
Oct 29 23:08:25.868356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:25.868328    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2"
Oct 29 23:08:25.981147 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:25.981109    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92ea570b-9973-4fd2-b18a-aa32192ba3d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92ea570b-9973-4fd2-b18a-aa32192ba3d8-plugins-config-volume\") pod \"92ea570b-9973-4fd2-b18a-aa32192ba3d8\" (UID: \"92ea570b-9973-4fd2-b18a-aa32192ba3d8\") "
Oct 29 23:08:25.981628 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:25.981345    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92ea570b-9973-4fd2-b18a-aa32192ba3d8-kube-api-access-bjr77\" (UniqueName: \"kubernetes.io/projected/92ea570b-9973-4fd2-b18a-aa32192ba3d8-kube-api-access-bjr77\") pod \"92ea570b-9973-4fd2-b18a-aa32192ba3d8\" (UID: \"92ea570b-9973-4fd2-b18a-aa32192ba3d8\") "
Oct 29 23:08:25.981628 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:25.981479    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ea570b-9973-4fd2-b18a-aa32192ba3d8-plugins-config-volume" pod "92ea570b-9973-4fd2-b18a-aa32192ba3d8" (UID: "92ea570b-9973-4fd2-b18a-aa32192ba3d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:25.981979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:25.981958    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92ea570b-9973-4fd2-b18a-aa32192ba3d8-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:25.983764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:25.983738    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ea570b-9973-4fd2-b18a-aa32192ba3d8-kube-api-access-bjr77" pod "92ea570b-9973-4fd2-b18a-aa32192ba3d8" (UID: "92ea570b-9973-4fd2-b18a-aa32192ba3d8"). InnerVolumeSpecName "kube-api-access-bjr77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:26.082840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.082748    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjr77\" (UniqueName: \"kubernetes.io/projected/92ea570b-9973-4fd2-b18a-aa32192ba3d8-kube-api-access-bjr77\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:26.122434 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.122401    5428 generic.go:356] "Generic (PLEG): container finished" podID="92ea570b-9973-4fd2-b18a-aa32192ba3d8" containerID="9135a195457e41d388b1a228ab52fbad7c2dfe7f82a52adc1178c20a17c42acf" exitCode=0
Oct 29 23:08:26.122599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.122446    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2"
Oct 29 23:08:26.122599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.122486    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2" event={"ID":"92ea570b-9973-4fd2-b18a-aa32192ba3d8","Type":"ContainerDied","Data":"9135a195457e41d388b1a228ab52fbad7c2dfe7f82a52adc1178c20a17c42acf"}
Oct 29 23:08:26.122599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.122544    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2" event={"ID":"92ea570b-9973-4fd2-b18a-aa32192ba3d8","Type":"ContainerDied","Data":"6d2cdd2e10295139ca2059866c1df3966d6be4b5162bbb06a18daf51b81f8038"}
Oct 29 23:08:26.122599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.122571    5428 scope.go:122] "RemoveContainer" containerID="9135a195457e41d388b1a228ab52fbad7c2dfe7f82a52adc1178c20a17c42acf"
Oct 29 23:08:26.133386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.133371    5428 scope.go:122] "RemoveContainer" containerID="9135a195457e41d388b1a228ab52fbad7c2dfe7f82a52adc1178c20a17c42acf"
Oct 29 23:08:26.133660 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:08:26.133641    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9135a195457e41d388b1a228ab52fbad7c2dfe7f82a52adc1178c20a17c42acf\": container with ID starting with 9135a195457e41d388b1a228ab52fbad7c2dfe7f82a52adc1178c20a17c42acf not found: ID does not exist" containerID="9135a195457e41d388b1a228ab52fbad7c2dfe7f82a52adc1178c20a17c42acf"
Oct 29 23:08:26.133707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.133667    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9135a195457e41d388b1a228ab52fbad7c2dfe7f82a52adc1178c20a17c42acf"} err="failed to get container status \"9135a195457e41d388b1a228ab52fbad7c2dfe7f82a52adc1178c20a17c42acf\": rpc error: code = NotFound desc = could not find container \"9135a195457e41d388b1a228ab52fbad7c2dfe7f82a52adc1178c20a17c42acf\": container with ID starting with 9135a195457e41d388b1a228ab52fbad7c2dfe7f82a52adc1178c20a17c42acf not found: ID does not exist"
Oct 29 23:08:26.175416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.175387    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2"]
Oct 29 23:08:26.186809 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.186775    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-957eef-f5d5fdbdd-kp8c2"]
Oct 29 23:08:26.272530 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.272490    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2" podUID="44b8ab25-5d40-45fc-8a55-9ce13bcf8646" containerName="payload-processing" containerID="cri-o://c896946be39f6d394910c09be61848d09757e19d3475271ae3912a6af5b510c7" gracePeriod=25
Oct 29 23:08:26.535385 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.535357    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2"
Oct 29 23:08:26.589489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.589451    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b8ab25-5d40-45fc-8a55-9ce13bcf8646-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b8ab25-5d40-45fc-8a55-9ce13bcf8646-plugins-config-volume\") pod \"44b8ab25-5d40-45fc-8a55-9ce13bcf8646\" (UID: \"44b8ab25-5d40-45fc-8a55-9ce13bcf8646\") "
Oct 29 23:08:26.589773 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.589752    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b8ab25-5d40-45fc-8a55-9ce13bcf8646-kube-api-access-s62ks\" (UniqueName: \"kubernetes.io/projected/44b8ab25-5d40-45fc-8a55-9ce13bcf8646-kube-api-access-s62ks\") pod \"44b8ab25-5d40-45fc-8a55-9ce13bcf8646\" (UID: \"44b8ab25-5d40-45fc-8a55-9ce13bcf8646\") "
Oct 29 23:08:26.589873 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.589850    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b8ab25-5d40-45fc-8a55-9ce13bcf8646-plugins-config-volume" pod "44b8ab25-5d40-45fc-8a55-9ce13bcf8646" (UID: "44b8ab25-5d40-45fc-8a55-9ce13bcf8646"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:26.590452 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.590423    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b8ab25-5d40-45fc-8a55-9ce13bcf8646-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:26.592160 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.592126    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b8ab25-5d40-45fc-8a55-9ce13bcf8646-kube-api-access-s62ks" pod "44b8ab25-5d40-45fc-8a55-9ce13bcf8646" (UID: "44b8ab25-5d40-45fc-8a55-9ce13bcf8646"). InnerVolumeSpecName "kube-api-access-s62ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:26.691865 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.691756    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s62ks\" (UniqueName: \"kubernetes.io/projected/44b8ab25-5d40-45fc-8a55-9ce13bcf8646-kube-api-access-s62ks\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:26.712600 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.712564    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:26.779111 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.779079    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92ea570b-9973-4fd2-b18a-aa32192ba3d8" path="/var/lib/kubelet/pods/92ea570b-9973-4fd2-b18a-aa32192ba3d8/volumes"
Oct 29 23:08:26.845837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:26.845793    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:27.131567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:27.131534    5428 generic.go:356] "Generic (PLEG): container finished" podID="44b8ab25-5d40-45fc-8a55-9ce13bcf8646" containerID="c896946be39f6d394910c09be61848d09757e19d3475271ae3912a6af5b510c7" exitCode=0
Oct 29 23:08:27.132076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:27.131594    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2"
Oct 29 23:08:27.132076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:27.131624    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2" event={"ID":"44b8ab25-5d40-45fc-8a55-9ce13bcf8646","Type":"ContainerDied","Data":"c896946be39f6d394910c09be61848d09757e19d3475271ae3912a6af5b510c7"}
Oct 29 23:08:27.132076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:27.131691    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2" event={"ID":"44b8ab25-5d40-45fc-8a55-9ce13bcf8646","Type":"ContainerDied","Data":"05251e6507eb01a66567c2660d64a03d4e5708f45015c5801fb0f319dda4e8fa"}
Oct 29 23:08:27.132076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:27.131728    5428 scope.go:122] "RemoveContainer" containerID="c896946be39f6d394910c09be61848d09757e19d3475271ae3912a6af5b510c7"
Oct 29 23:08:27.142197 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:27.142176    5428 scope.go:122] "RemoveContainer" containerID="c896946be39f6d394910c09be61848d09757e19d3475271ae3912a6af5b510c7"
Oct 29 23:08:27.142468 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:08:27.142447    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c896946be39f6d394910c09be61848d09757e19d3475271ae3912a6af5b510c7\": container with ID starting with c896946be39f6d394910c09be61848d09757e19d3475271ae3912a6af5b510c7 not found: ID does not exist" containerID="c896946be39f6d394910c09be61848d09757e19d3475271ae3912a6af5b510c7"
Oct 29 23:08:27.142521 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:27.142476    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c896946be39f6d394910c09be61848d09757e19d3475271ae3912a6af5b510c7"} err="failed to get container status \"c896946be39f6d394910c09be61848d09757e19d3475271ae3912a6af5b510c7\": rpc error: code = NotFound desc = could not find container \"c896946be39f6d394910c09be61848d09757e19d3475271ae3912a6af5b510c7\": container with ID starting with c896946be39f6d394910c09be61848d09757e19d3475271ae3912a6af5b510c7 not found: ID does not exist"
Oct 29 23:08:27.187099 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:27.187074    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2"]
Oct 29 23:08:27.192503 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:27.192479    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ce5978-669979c946-dkqb2"]
Oct 29 23:08:27.495451 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:27.495354    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:27.848613 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:27.848541    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:27.892120 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:27.892079    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd"]
Oct 29 23:08:27.970513 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:27.970479    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:28.098545 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:28.098510    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:28.208064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:28.208029    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:28.778301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:28.778268    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b8ab25-5d40-45fc-8a55-9ce13bcf8646" path="/var/lib/kubelet/pods/44b8ab25-5d40-45fc-8a55-9ce13bcf8646/volumes"
Oct 29 23:08:29.177352 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.177298    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:29.569877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.569714    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:29.703844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.703768    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:29.712661 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.712626    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg"]
Oct 29 23:08:29.714312 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714284    5428 cpu_manager.go:414] "containerMap: removing container" podUID="82514398-fdc2-46df-b04f-9d5429af45e8" containerName="payload-processing"
Oct 29 23:08:29.714312 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714311    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82514398-fdc2-46df-b04f-9d5429af45e8" containerName="payload-processing"
Oct 29 23:08:29.714648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714338    5428 cpu_manager.go:414] "containerMap: removing container" podUID="d5294cfe-5196-4b3b-bfbe-60afe05d6cb5" containerName="payload-pre-processing"
Oct 29 23:08:29.714648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714347    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5294cfe-5196-4b3b-bfbe-60afe05d6cb5" containerName="payload-pre-processing"
Oct 29 23:08:29.714648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714371    5428 cpu_manager.go:414] "containerMap: removing container" podUID="c946b973-a30a-432c-8ee0-5ea240c5ee26" containerName="payload-pre-processing"
Oct 29 23:08:29.714648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714380    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c946b973-a30a-432c-8ee0-5ea240c5ee26" containerName="payload-pre-processing"
Oct 29 23:08:29.714648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714412    5428 cpu_manager.go:414] "containerMap: removing container" podUID="d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" containerName="istio-proxy"
Oct 29 23:08:29.714648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714419    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" containerName="istio-proxy"
Oct 29 23:08:29.714648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714439    5428 cpu_manager.go:414] "containerMap: removing container" podUID="53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" containerName="istio-proxy"
Oct 29 23:08:29.714648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714446    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" containerName="istio-proxy"
Oct 29 23:08:29.714648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714470    5428 cpu_manager.go:414] "containerMap: removing container" podUID="847d1425-c094-4abc-b085-55d7c9c2bcbd" containerName="payload-processing"
Oct 29 23:08:29.714648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714478    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847d1425-c094-4abc-b085-55d7c9c2bcbd" containerName="payload-processing"
Oct 29 23:08:29.714648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714509    5428 cpu_manager.go:414] "containerMap: removing container" podUID="92ea570b-9973-4fd2-b18a-aa32192ba3d8" containerName="payload-processing"
Oct 29 23:08:29.714648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714516    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ea570b-9973-4fd2-b18a-aa32192ba3d8" containerName="payload-processing"
Oct 29 23:08:29.714648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714565    5428 cpu_manager.go:414] "containerMap: removing container" podUID="44b8ab25-5d40-45fc-8a55-9ce13bcf8646" containerName="payload-processing"
Oct 29 23:08:29.714648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714573    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b8ab25-5d40-45fc-8a55-9ce13bcf8646" containerName="payload-processing"
Oct 29 23:08:29.714648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714586    5428 cpu_manager.go:414] "containerMap: removing container" podUID="cb5221e1-3731-4150-aaba-89507c4b0cc8" containerName="maas-api"
Oct 29 23:08:29.714648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714592    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb5221e1-3731-4150-aaba-89507c4b0cc8" containerName="maas-api"
Oct 29 23:08:29.716220 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.714994    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="847d1425-c094-4abc-b085-55d7c9c2bcbd" containerName="payload-processing"
Oct 29 23:08:29.716220 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.715014    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ea570b-9973-4fd2-b18a-aa32192ba3d8" containerName="payload-processing"
Oct 29 23:08:29.716220 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.715039    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b8ab25-5d40-45fc-8a55-9ce13bcf8646" containerName="payload-processing"
Oct 29 23:08:29.716220 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.715057    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb5221e1-3731-4150-aaba-89507c4b0cc8" containerName="maas-api"
Oct 29 23:08:29.716220 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.715084    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="82514398-fdc2-46df-b04f-9d5429af45e8" containerName="payload-processing"
Oct 29 23:08:29.716220 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.715101    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5294cfe-5196-4b3b-bfbe-60afe05d6cb5" containerName="payload-pre-processing"
Oct 29 23:08:29.716220 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.715109    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c946b973-a30a-432c-8ee0-5ea240c5ee26" containerName="payload-pre-processing"
Oct 29 23:08:29.716220 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.715128    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85de9d4-72c8-4340-ae0d-f9c8735f5a4f" containerName="istio-proxy"
Oct 29 23:08:29.716220 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.715144    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e3c3eb-c2d6-4c8d-b1db-c953a8964b28" containerName="istio-proxy"
Oct 29 23:08:29.723867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.723843    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg"
Oct 29 23:08:29.748344 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.748238    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg"]
Oct 29 23:08:29.796894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.796866    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-31d798\""
Oct 29 23:08:29.797137 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.796873    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-31d798-dockercfg-g6f69\""
Oct 29 23:08:29.837380 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.837291    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg7lp\" (UniqueName: \"kubernetes.io/projected/85bd1ee8-788b-492f-a1ed-e09be149f040-kube-api-access-mg7lp\") pod \"payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg\" (UID: \"85bd1ee8-788b-492f-a1ed-e09be149f040\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg"
Oct 29 23:08:29.837648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.837614    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85bd1ee8-788b-492f-a1ed-e09be149f040-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg\" (UID: \"85bd1ee8-788b-492f-a1ed-e09be149f040\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg"
Oct 29 23:08:29.939362 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.939319    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85bd1ee8-788b-492f-a1ed-e09be149f040-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg\" (UID: \"85bd1ee8-788b-492f-a1ed-e09be149f040\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg"
Oct 29 23:08:29.939611 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.939387    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg7lp\" (UniqueName: \"kubernetes.io/projected/85bd1ee8-788b-492f-a1ed-e09be149f040-kube-api-access-mg7lp\") pod \"payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg\" (UID: \"85bd1ee8-788b-492f-a1ed-e09be149f040\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg"
Oct 29 23:08:29.940289 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.940264    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85bd1ee8-788b-492f-a1ed-e09be149f040-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg\" (UID: \"85bd1ee8-788b-492f-a1ed-e09be149f040\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg"
Oct 29 23:08:29.963065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:29.963027    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg7lp\" (UniqueName: \"kubernetes.io/projected/85bd1ee8-788b-492f-a1ed-e09be149f040-kube-api-access-mg7lp\") pod \"payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg\" (UID: \"85bd1ee8-788b-492f-a1ed-e09be149f040\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg"
Oct 29 23:08:30.051033 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:30.050987    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg"
Oct 29 23:08:30.392208 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:30.375029    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg"]
Oct 29 23:08:30.392208 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:08:30.377756    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bd1ee8_788b_492f_a1ed_e09be149f040.slice/crio-be45d271f8939d0d8ee1ab3d895bda4dfa641c56c8993bc3ced2c94baac4b317 WatchSource:0}: Error finding container be45d271f8939d0d8ee1ab3d895bda4dfa641c56c8993bc3ced2c94baac4b317: Status 404 returned error can't find the container with id be45d271f8939d0d8ee1ab3d895bda4dfa641c56c8993bc3ced2c94baac4b317
Oct 29 23:08:30.729665 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:30.729565    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"]
Oct 29 23:08:30.730164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:30.729820    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j" podUID="3bca0741-fd7f-4183-b9f7-f5c827532f72" containerName="maas-api" containerID="cri-o://17529fab1dbfb8332ddc381b82330d3fed385feb6ac37c8c5173146ca3af4242" gracePeriod=30
Oct 29 23:08:30.809930 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:30.809882    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:31.047789 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.047762    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:08:31.173115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.173078    5428 generic.go:356] "Generic (PLEG): container finished" podID="3bca0741-fd7f-4183-b9f7-f5c827532f72" containerID="17529fab1dbfb8332ddc381b82330d3fed385feb6ac37c8c5173146ca3af4242" exitCode=0
Oct 29 23:08:31.173286 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.173146    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"
Oct 29 23:08:31.173286 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.173166    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j" event={"ID":"3bca0741-fd7f-4183-b9f7-f5c827532f72","Type":"ContainerDied","Data":"17529fab1dbfb8332ddc381b82330d3fed385feb6ac37c8c5173146ca3af4242"}
Oct 29 23:08:31.173286 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.173224    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j" event={"ID":"3bca0741-fd7f-4183-b9f7-f5c827532f72","Type":"ContainerDied","Data":"ebd0b6f18b9faa43d22d2a689fff56d7bdb227bff26061ba0da51945e69eaaa9"}
Oct 29 23:08:31.173286 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.173256    5428 scope.go:122] "RemoveContainer" containerID="17529fab1dbfb8332ddc381b82330d3fed385feb6ac37c8c5173146ca3af4242"
Oct 29 23:08:31.173542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.173518    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bca0741-fd7f-4183-b9f7-f5c827532f72-kube-api-access-l4pd9\" (UniqueName: \"kubernetes.io/projected/3bca0741-fd7f-4183-b9f7-f5c827532f72-kube-api-access-l4pd9\") pod \"3bca0741-fd7f-4183-b9f7-f5c827532f72\" (UID: \"3bca0741-fd7f-4183-b9f7-f5c827532f72\") "
Oct 29 23:08:31.173654 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.173634    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bca0741-fd7f-4183-b9f7-f5c827532f72-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bca0741-fd7f-4183-b9f7-f5c827532f72-metrics-certs\") pod \"3bca0741-fd7f-4183-b9f7-f5c827532f72\" (UID: \"3bca0741-fd7f-4183-b9f7-f5c827532f72\") "
Oct 29 23:08:31.173720 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.173689    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bca0741-fd7f-4183-b9f7-f5c827532f72-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bca0741-fd7f-4183-b9f7-f5c827532f72-openshift-service-ca\") pod \"3bca0741-fd7f-4183-b9f7-f5c827532f72\" (UID: \"3bca0741-fd7f-4183-b9f7-f5c827532f72\") "
Oct 29 23:08:31.173773 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.173715    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bca0741-fd7f-4183-b9f7-f5c827532f72-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bca0741-fd7f-4183-b9f7-f5c827532f72-maas-api-tls\") pod \"3bca0741-fd7f-4183-b9f7-f5c827532f72\" (UID: \"3bca0741-fd7f-4183-b9f7-f5c827532f72\") "
Oct 29 23:08:31.174427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.174399    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bca0741-fd7f-4183-b9f7-f5c827532f72-openshift-service-ca" pod "3bca0741-fd7f-4183-b9f7-f5c827532f72" (UID: "3bca0741-fd7f-4183-b9f7-f5c827532f72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:31.175365 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.175328    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg" event={"ID":"85bd1ee8-788b-492f-a1ed-e09be149f040","Type":"ContainerStarted","Data":"760b70ebd3b42e6bcd7472e437e5ec4db80bc4dc056f4b35ec4a3e49dd14553e"}
Oct 29 23:08:31.175473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.175407    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg" event={"ID":"85bd1ee8-788b-492f-a1ed-e09be149f040","Type":"ContainerStarted","Data":"be45d271f8939d0d8ee1ab3d895bda4dfa641c56c8993bc3ced2c94baac4b317"}
Oct 29 23:08:31.175473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.175443    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg"
Oct 29 23:08:31.177983 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.177960    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bca0741-fd7f-4183-b9f7-f5c827532f72-kube-api-access-l4pd9" pod "3bca0741-fd7f-4183-b9f7-f5c827532f72" (UID: "3bca0741-fd7f-4183-b9f7-f5c827532f72"). InnerVolumeSpecName "kube-api-access-l4pd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:31.178478 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.178442    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bca0741-fd7f-4183-b9f7-f5c827532f72-maas-api-tls" pod "3bca0741-fd7f-4183-b9f7-f5c827532f72" (UID: "3bca0741-fd7f-4183-b9f7-f5c827532f72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:31.187330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.187244    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bca0741-fd7f-4183-b9f7-f5c827532f72-metrics-certs" pod "3bca0741-fd7f-4183-b9f7-f5c827532f72" (UID: "3bca0741-fd7f-4183-b9f7-f5c827532f72"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:31.220876 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.220848    5428 scope.go:122] "RemoveContainer" containerID="17529fab1dbfb8332ddc381b82330d3fed385feb6ac37c8c5173146ca3af4242"
Oct 29 23:08:31.221278 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:08:31.221251    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17529fab1dbfb8332ddc381b82330d3fed385feb6ac37c8c5173146ca3af4242\": container with ID starting with 17529fab1dbfb8332ddc381b82330d3fed385feb6ac37c8c5173146ca3af4242 not found: ID does not exist" containerID="17529fab1dbfb8332ddc381b82330d3fed385feb6ac37c8c5173146ca3af4242"
Oct 29 23:08:31.221351 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.221292    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17529fab1dbfb8332ddc381b82330d3fed385feb6ac37c8c5173146ca3af4242"} err="failed to get container status \"17529fab1dbfb8332ddc381b82330d3fed385feb6ac37c8c5173146ca3af4242\": rpc error: code = NotFound desc = could not find container \"17529fab1dbfb8332ddc381b82330d3fed385feb6ac37c8c5173146ca3af4242\": container with ID starting with 17529fab1dbfb8332ddc381b82330d3fed385feb6ac37c8c5173146ca3af4242 not found: ID does not exist"
Oct 29 23:08:31.275856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.275779    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4pd9\" (UniqueName: \"kubernetes.io/projected/3bca0741-fd7f-4183-b9f7-f5c827532f72-kube-api-access-l4pd9\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:31.275856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.275823    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bca0741-fd7f-4183-b9f7-f5c827532f72-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:31.275856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.275838    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bca0741-fd7f-4183-b9f7-f5c827532f72-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:31.275856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.275852    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bca0741-fd7f-4183-b9f7-f5c827532f72-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:31.518718 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.518659    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg" podStartSLOduration=2.51864504 podStartE2EDuration="2.51864504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:31.234005987 +0000 UTC m=+1514.576954736" watchObservedRunningTime="2026-10-29 23:08:31.51864504 +0000 UTC m=+1514.861593831"
Oct 29 23:08:31.521528 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.521501    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"]
Oct 29 23:08:31.534631 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.534568    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ce5978-76987898d9-zns9j"]
Oct 29 23:08:31.544759 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.544731    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:31.700576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.700544    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:31.965899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:31.965868    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:32.246764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:32.246676    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:32.779436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:32.779404    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bca0741-fd7f-4183-b9f7-f5c827532f72" path="/var/lib/kubelet/pods/3bca0741-fd7f-4183-b9f7-f5c827532f72/volumes"
Oct 29 23:08:32.893368 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:32.893331    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd" podUID="76880e88-0196-4501-b499-52a0800e26dc" containerName="payload-processing" containerID="cri-o://b50b0d3f7b1223fe2bef44051f30b5116877359b5a40c9261fcc8afe7f856dd4" gracePeriod=25
Oct 29 23:08:33.185608 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:33.185583    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd"
Oct 29 23:08:33.195260 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:33.195232    5428 generic.go:356] "Generic (PLEG): container finished" podID="76880e88-0196-4501-b499-52a0800e26dc" containerID="b50b0d3f7b1223fe2bef44051f30b5116877359b5a40c9261fcc8afe7f856dd4" exitCode=0
Oct 29 23:08:33.195396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:33.195275    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd"
Oct 29 23:08:33.195396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:33.195329    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd" event={"ID":"76880e88-0196-4501-b499-52a0800e26dc","Type":"ContainerDied","Data":"b50b0d3f7b1223fe2bef44051f30b5116877359b5a40c9261fcc8afe7f856dd4"}
Oct 29 23:08:33.195396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:33.195375    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd" event={"ID":"76880e88-0196-4501-b499-52a0800e26dc","Type":"ContainerDied","Data":"dc39bfa107aecae7e0f25cce87bc3102f83782575a4e4a01bcebe9733091b271"}
Oct 29 23:08:33.195527 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:33.195402    5428 scope.go:122] "RemoveContainer" containerID="b50b0d3f7b1223fe2bef44051f30b5116877359b5a40c9261fcc8afe7f856dd4"
Oct 29 23:08:33.205644 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:33.205625    5428 scope.go:122] "RemoveContainer" containerID="b50b0d3f7b1223fe2bef44051f30b5116877359b5a40c9261fcc8afe7f856dd4"
Oct 29 23:08:33.205975 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:08:33.205947    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b50b0d3f7b1223fe2bef44051f30b5116877359b5a40c9261fcc8afe7f856dd4\": container with ID starting with b50b0d3f7b1223fe2bef44051f30b5116877359b5a40c9261fcc8afe7f856dd4 not found: ID does not exist" containerID="b50b0d3f7b1223fe2bef44051f30b5116877359b5a40c9261fcc8afe7f856dd4"
Oct 29 23:08:33.206040 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:33.205982    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b50b0d3f7b1223fe2bef44051f30b5116877359b5a40c9261fcc8afe7f856dd4"} err="failed to get container status \"b50b0d3f7b1223fe2bef44051f30b5116877359b5a40c9261fcc8afe7f856dd4\": rpc error: code = NotFound desc = could not find container \"b50b0d3f7b1223fe2bef44051f30b5116877359b5a40c9261fcc8afe7f856dd4\": container with ID starting with b50b0d3f7b1223fe2bef44051f30b5116877359b5a40c9261fcc8afe7f856dd4 not found: ID does not exist"
Oct 29 23:08:33.301753 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:33.301720    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76880e88-0196-4501-b499-52a0800e26dc-kube-api-access-5kxjw\" (UniqueName: \"kubernetes.io/projected/76880e88-0196-4501-b499-52a0800e26dc-kube-api-access-5kxjw\") pod \"76880e88-0196-4501-b499-52a0800e26dc\" (UID: \"76880e88-0196-4501-b499-52a0800e26dc\") "
Oct 29 23:08:33.301971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:33.301826    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76880e88-0196-4501-b499-52a0800e26dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76880e88-0196-4501-b499-52a0800e26dc-plugins-config-volume\") pod \"76880e88-0196-4501-b499-52a0800e26dc\" (UID: \"76880e88-0196-4501-b499-52a0800e26dc\") "
Oct 29 23:08:33.302330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:33.302302    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76880e88-0196-4501-b499-52a0800e26dc-plugins-config-volume" pod "76880e88-0196-4501-b499-52a0800e26dc" (UID: "76880e88-0196-4501-b499-52a0800e26dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:33.302448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:33.302422    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76880e88-0196-4501-b499-52a0800e26dc-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:33.304636 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:33.304602    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76880e88-0196-4501-b499-52a0800e26dc-kube-api-access-5kxjw" pod "76880e88-0196-4501-b499-52a0800e26dc" (UID: "76880e88-0196-4501-b499-52a0800e26dc"). InnerVolumeSpecName "kube-api-access-5kxjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:33.403263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:33.403239    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kxjw\" (UniqueName: \"kubernetes.io/projected/76880e88-0196-4501-b499-52a0800e26dc-kube-api-access-5kxjw\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:33.568987 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:33.568947    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd"]
Oct 29 23:08:33.582671 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:33.582642    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dfedfe-f7fcf556d-pjckd"]
Oct 29 23:08:33.635474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:33.635432    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:34.545070 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:34.545024    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:34.665758 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:34.665712    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:34.784968 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:34.784931    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:34.787582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:34.787558    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76880e88-0196-4501-b499-52a0800e26dc" path="/var/lib/kubelet/pods/76880e88-0196-4501-b499-52a0800e26dc/volumes"
Oct 29 23:08:35.202465 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:35.202432    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:37.297785 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:37.297742    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:39.272832 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:39.271726    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:39.535411 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:39.535321    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:39.719823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:39.719772    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:39.836914 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:39.836824    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:41.475925 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:41.475893    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:41.720485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:41.720447    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:41.835283 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:41.835200    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:42.056096 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:42.056061    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:42.189580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:42.189552    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg"
Oct 29 23:08:43.573367 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:43.573336    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:45.003881 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.003840    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"]
Oct 29 23:08:45.004369 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.004135    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk" podUID="2753c078-4e19-40c1-b4f0-c1591c6d64d4" containerName="maas-api" containerID="cri-o://e305de0f1c917926a232c84fe67854c09e3c135468685c47687b981cfd685ba2" gracePeriod=30
Oct 29 23:08:45.071959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.071930    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:45.294113 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.293993    5428 generic.go:356] "Generic (PLEG): container finished" podID="2753c078-4e19-40c1-b4f0-c1591c6d64d4" containerID="e305de0f1c917926a232c84fe67854c09e3c135468685c47687b981cfd685ba2" exitCode=0
Oct 29 23:08:45.294113 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.294055    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk" event={"ID":"2753c078-4e19-40c1-b4f0-c1591c6d64d4","Type":"ContainerDied","Data":"e305de0f1c917926a232c84fe67854c09e3c135468685c47687b981cfd685ba2"}
Oct 29 23:08:45.339704 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.339676    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"
Oct 29 23:08:45.438305 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.438271    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2753c078-4e19-40c1-b4f0-c1591c6d64d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2753c078-4e19-40c1-b4f0-c1591c6d64d4-maas-api-tls\") pod \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\" (UID: \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\") "
Oct 29 23:08:45.438548 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.438485    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2753c078-4e19-40c1-b4f0-c1591c6d64d4-kube-api-access-5b7c7\" (UniqueName: \"kubernetes.io/projected/2753c078-4e19-40c1-b4f0-c1591c6d64d4-kube-api-access-5b7c7\") pod \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\" (UID: \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\") "
Oct 29 23:08:45.439503 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.439473    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2753c078-4e19-40c1-b4f0-c1591c6d64d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2753c078-4e19-40c1-b4f0-c1591c6d64d4-metrics-certs\") pod \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\" (UID: \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\") "
Oct 29 23:08:45.439645 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.439524    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2753c078-4e19-40c1-b4f0-c1591c6d64d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2753c078-4e19-40c1-b4f0-c1591c6d64d4-openshift-service-ca\") pod \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\" (UID: \"2753c078-4e19-40c1-b4f0-c1591c6d64d4\") "
Oct 29 23:08:45.440415 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.440368    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2753c078-4e19-40c1-b4f0-c1591c6d64d4-openshift-service-ca" pod "2753c078-4e19-40c1-b4f0-c1591c6d64d4" (UID: "2753c078-4e19-40c1-b4f0-c1591c6d64d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:45.441288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.441253    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2753c078-4e19-40c1-b4f0-c1591c6d64d4-maas-api-tls" pod "2753c078-4e19-40c1-b4f0-c1591c6d64d4" (UID: "2753c078-4e19-40c1-b4f0-c1591c6d64d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:45.442617 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.442580    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2753c078-4e19-40c1-b4f0-c1591c6d64d4-metrics-certs" pod "2753c078-4e19-40c1-b4f0-c1591c6d64d4" (UID: "2753c078-4e19-40c1-b4f0-c1591c6d64d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:45.442728 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.442704    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2753c078-4e19-40c1-b4f0-c1591c6d64d4-kube-api-access-5b7c7" pod "2753c078-4e19-40c1-b4f0-c1591c6d64d4" (UID: "2753c078-4e19-40c1-b4f0-c1591c6d64d4"). InnerVolumeSpecName "kube-api-access-5b7c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:45.541770 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.541725    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2753c078-4e19-40c1-b4f0-c1591c6d64d4-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:45.541770 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.541774    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2753c078-4e19-40c1-b4f0-c1591c6d64d4-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:45.541998 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.541793    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2753c078-4e19-40c1-b4f0-c1591c6d64d4-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:45.541998 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.541827    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b7c7\" (UniqueName: \"kubernetes.io/projected/2753c078-4e19-40c1-b4f0-c1591c6d64d4-kube-api-access-5b7c7\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:45.932898 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:45.932867    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:46.061098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:46.061062    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:46.169877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:46.169839    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:46.302714 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:46.302627    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"
Oct 29 23:08:46.302714 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:46.302659    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk" event={"ID":"2753c078-4e19-40c1-b4f0-c1591c6d64d4","Type":"ContainerDied","Data":"65792026dbd4519a0be96465077869f5c6c1cbe45b8374048cbbd55b6cd5b50c"}
Oct 29 23:08:46.302966 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:46.302730    5428 scope.go:122] "RemoveContainer" containerID="e305de0f1c917926a232c84fe67854c09e3c135468685c47687b981cfd685ba2"
Oct 29 23:08:46.356026 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:46.355994    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"]
Oct 29 23:08:46.368475 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:46.368445    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-957eef-54f965d6d6-v7zrk"]
Oct 29 23:08:46.573540 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:46.573467    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:46.779136 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:46.779105    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2753c078-4e19-40c1-b4f0-c1591c6d64d4" path="/var/lib/kubelet/pods/2753c078-4e19-40c1-b4f0-c1591c6d64d4/volumes"
Oct 29 23:08:47.367949 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:08:47.367910    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:08:48.712935 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:48.712899    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:50.358863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:50.358826    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:50.684647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:50.684614    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f"]
Oct 29 23:08:50.795176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:50.795146    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:50.913720 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:50.913692    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:52.182378 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:52.182345    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:52.970489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:52.970454    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:53.041579 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:53.041542    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98"]
Oct 29 23:08:53.074784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:53.074749    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z"]
Oct 29 23:08:53.089565 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:53.089532    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:53.205196 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:53.205156    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:53.318864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:53.318683    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:53.458071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:53.458037    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:54.284630 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:54.284597    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:55.685466 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:55.685412    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f" podUID="f8b78432-161a-418b-aeaf-a0580dc80b54" containerName="payload-pre-processing" containerID="cri-o://f5af08cccd903e72b518a5a66a675f7564afd562c1626719c58d78d474e5b61d" gracePeriod=25
Oct 29 23:08:55.704662 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:55.704627    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"]
Oct 29 23:08:55.705223 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:55.705149    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8" podUID="a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7" containerName="maas-api" containerID="cri-o://fc40ff8138d775f46cd7579a7c4e94c89615a314be9cfb08dbca812e29e9794a" gracePeriod=30
Oct 29 23:08:55.734237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:55.734108    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"]
Oct 29 23:08:55.734422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:55.734389    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x" podUID="75a16c52-71c1-4da1-bc73-4e6fea6184c8" containerName="istio-proxy" containerID="cri-o://f5af76081a2c09f4a2b14c5fda368876ea76b3b70c9098a3bd55340fa656711e" gracePeriod=30
Oct 29 23:08:55.773013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:55.772982    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:56.170857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.169115    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f"
Oct 29 23:08:56.176867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.174760    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"
Oct 29 23:08:56.250904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.247361    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-maas-api-tls\") pod \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\" (UID: \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\") "
Oct 29 23:08:56.250904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.247524    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-kube-api-access-ng6sm\" (UniqueName: \"kubernetes.io/projected/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-kube-api-access-ng6sm\") pod \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\" (UID: \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\") "
Oct 29 23:08:56.250904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.247573    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8b78432-161a-418b-aeaf-a0580dc80b54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b78432-161a-418b-aeaf-a0580dc80b54-plugins-config-volume\") pod \"f8b78432-161a-418b-aeaf-a0580dc80b54\" (UID: \"f8b78432-161a-418b-aeaf-a0580dc80b54\") "
Oct 29 23:08:56.250904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.247633    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-metrics-certs\") pod \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\" (UID: \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\") "
Oct 29 23:08:56.250904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.247886    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b78432-161a-418b-aeaf-a0580dc80b54-kube-api-access-59xs8\" (UniqueName: \"kubernetes.io/projected/f8b78432-161a-418b-aeaf-a0580dc80b54-kube-api-access-59xs8\") pod \"f8b78432-161a-418b-aeaf-a0580dc80b54\" (UID: \"f8b78432-161a-418b-aeaf-a0580dc80b54\") "
Oct 29 23:08:56.250904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.247927    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-openshift-service-ca\") pod \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\" (UID: \"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7\") "
Oct 29 23:08:56.250904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.249491    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b78432-161a-418b-aeaf-a0580dc80b54-plugins-config-volume" pod "f8b78432-161a-418b-aeaf-a0580dc80b54" (UID: "f8b78432-161a-418b-aeaf-a0580dc80b54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:56.250904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.250096    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-openshift-service-ca" pod "a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7" (UID: "a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:56.266375 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.265976    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-metrics-certs" pod "a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7" (UID: "a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:56.266375 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.266000    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-kube-api-access-ng6sm" pod "a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7" (UID: "a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7"). InnerVolumeSpecName "kube-api-access-ng6sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:56.266375 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.266057    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b78432-161a-418b-aeaf-a0580dc80b54-kube-api-access-59xs8" pod "f8b78432-161a-418b-aeaf-a0580dc80b54" (UID: "f8b78432-161a-418b-aeaf-a0580dc80b54"). InnerVolumeSpecName "kube-api-access-59xs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:56.266375 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.266222    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-maas-api-tls" pod "a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7" (UID: "a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:56.364409 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.364368    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng6sm\" (UniqueName: \"kubernetes.io/projected/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-kube-api-access-ng6sm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:56.364609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.364415    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b78432-161a-418b-aeaf-a0580dc80b54-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:56.364609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.364433    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:56.364609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.364451    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59xs8\" (UniqueName: \"kubernetes.io/projected/f8b78432-161a-418b-aeaf-a0580dc80b54-kube-api-access-59xs8\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:56.364609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.364467    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:56.364609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.364486    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:56.388460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.388418    5428 generic.go:356] "Generic (PLEG): container finished" podID="a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7" containerID="fc40ff8138d775f46cd7579a7c4e94c89615a314be9cfb08dbca812e29e9794a" exitCode=0
Oct 29 23:08:56.388683 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.388660    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8" event={"ID":"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7","Type":"ContainerDied","Data":"fc40ff8138d775f46cd7579a7c4e94c89615a314be9cfb08dbca812e29e9794a"}
Oct 29 23:08:56.390204 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.390169    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8" event={"ID":"a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7","Type":"ContainerDied","Data":"de86ac9e5da079f3a93079bbf7076d8af470ac6f80409f4159785f9c5ee16c59"}
Oct 29 23:08:56.390343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.390222    5428 scope.go:122] "RemoveContainer" containerID="fc40ff8138d775f46cd7579a7c4e94c89615a314be9cfb08dbca812e29e9794a"
Oct 29 23:08:56.390513 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.390425    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"
Oct 29 23:08:56.400827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.394861    5428 generic.go:356] "Generic (PLEG): container finished" podID="f8b78432-161a-418b-aeaf-a0580dc80b54" containerID="f5af08cccd903e72b518a5a66a675f7564afd562c1626719c58d78d474e5b61d" exitCode=0
Oct 29 23:08:56.400827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.395008    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f" event={"ID":"f8b78432-161a-418b-aeaf-a0580dc80b54","Type":"ContainerDied","Data":"f5af08cccd903e72b518a5a66a675f7564afd562c1626719c58d78d474e5b61d"}
Oct 29 23:08:56.400827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.395053    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f" event={"ID":"f8b78432-161a-418b-aeaf-a0580dc80b54","Type":"ContainerDied","Data":"3c3f7a3496a00f3815c164a8ca5169d8fd8a270fb1e3dd3610a35070e14976d2"}
Oct 29 23:08:56.400827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.395151    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f"
Oct 29 23:08:56.419377 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.419348    5428 scope.go:122] "RemoveContainer" containerID="fc40ff8138d775f46cd7579a7c4e94c89615a314be9cfb08dbca812e29e9794a"
Oct 29 23:08:56.419771 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:08:56.419739    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc40ff8138d775f46cd7579a7c4e94c89615a314be9cfb08dbca812e29e9794a\": container with ID starting with fc40ff8138d775f46cd7579a7c4e94c89615a314be9cfb08dbca812e29e9794a not found: ID does not exist" containerID="fc40ff8138d775f46cd7579a7c4e94c89615a314be9cfb08dbca812e29e9794a"
Oct 29 23:08:56.419916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.419782    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc40ff8138d775f46cd7579a7c4e94c89615a314be9cfb08dbca812e29e9794a"} err="failed to get container status \"fc40ff8138d775f46cd7579a7c4e94c89615a314be9cfb08dbca812e29e9794a\": rpc error: code = NotFound desc = could not find container \"fc40ff8138d775f46cd7579a7c4e94c89615a314be9cfb08dbca812e29e9794a\": container with ID starting with fc40ff8138d775f46cd7579a7c4e94c89615a314be9cfb08dbca812e29e9794a not found: ID does not exist"
Oct 29 23:08:56.419916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.419819    5428 scope.go:122] "RemoveContainer" containerID="f5af08cccd903e72b518a5a66a675f7564afd562c1626719c58d78d474e5b61d"
Oct 29 23:08:56.430646 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.430625    5428 scope.go:122] "RemoveContainer" containerID="f5af08cccd903e72b518a5a66a675f7564afd562c1626719c58d78d474e5b61d"
Oct 29 23:08:56.431090 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:08:56.431063    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5af08cccd903e72b518a5a66a675f7564afd562c1626719c58d78d474e5b61d\": container with ID starting with f5af08cccd903e72b518a5a66a675f7564afd562c1626719c58d78d474e5b61d not found: ID does not exist" containerID="f5af08cccd903e72b518a5a66a675f7564afd562c1626719c58d78d474e5b61d"
Oct 29 23:08:56.431173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.431102    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5af08cccd903e72b518a5a66a675f7564afd562c1626719c58d78d474e5b61d"} err="failed to get container status \"f5af08cccd903e72b518a5a66a675f7564afd562c1626719c58d78d474e5b61d\": rpc error: code = NotFound desc = could not find container \"f5af08cccd903e72b518a5a66a675f7564afd562c1626719c58d78d474e5b61d\": container with ID starting with f5af08cccd903e72b518a5a66a675f7564afd562c1626719c58d78d474e5b61d not found: ID does not exist"
Oct 29 23:08:56.459658 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.459614    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"]
Oct 29 23:08:56.463230 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.463201    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-57b28e-548fbc68c9-5clk8"]
Oct 29 23:08:56.503494 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.503409    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f"]
Oct 29 23:08:56.507392 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.507357    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cd000a-6c479fcbdc-v7s2f"]
Oct 29 23:08:56.767988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.767888    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:56.781985 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.781951    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7" path="/var/lib/kubelet/pods/a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7/volumes"
Oct 29 23:08:56.782491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:56.782474    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8b78432-161a-418b-aeaf-a0580dc80b54" path="/var/lib/kubelet/pods/f8b78432-161a-418b-aeaf-a0580dc80b54/volumes"
Oct 29 23:08:57.078413 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:57.078331    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:57.197468 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:57.196542    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:57.661585 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:57.661501    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:58.042839 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.042742    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98" podUID="fa734e34-458a-4633-9caf-c1e1b97f5c6f" containerName="payload-processing" containerID="cri-o://5f87f4332bdc78c191aecf3ac6cf77c6090decc442e7c010e619e823409b827a" gracePeriod=25
Oct 29 23:08:58.075477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.075443    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z" podUID="02f4bd43-6a70-471e-aa6c-459bdeaca665" containerName="payload-pre-processing" containerID="cri-o://b00eb59943c4265dc1de865f36186f3e3817c9d63b1fab108919301d376b40ae" gracePeriod=25
Oct 29 23:08:58.367682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.367652    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98"
Oct 29 23:08:58.414559 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.414528    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z"
Oct 29 23:08:58.414908 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.414873    5428 generic.go:356] "Generic (PLEG): container finished" podID="fa734e34-458a-4633-9caf-c1e1b97f5c6f" containerID="5f87f4332bdc78c191aecf3ac6cf77c6090decc442e7c010e619e823409b827a" exitCode=0
Oct 29 23:08:58.415035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.414905    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98" event={"ID":"fa734e34-458a-4633-9caf-c1e1b97f5c6f","Type":"ContainerDied","Data":"5f87f4332bdc78c191aecf3ac6cf77c6090decc442e7c010e619e823409b827a"}
Oct 29 23:08:58.415035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.414921    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98"
Oct 29 23:08:58.415035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.414960    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98" event={"ID":"fa734e34-458a-4633-9caf-c1e1b97f5c6f","Type":"ContainerDied","Data":"e75b5461aa6346255d1f29c3e65219c6b49ac5ef218a2397eb1cd7c9e7f0c017"}
Oct 29 23:08:58.415035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.414987    5428 scope.go:122] "RemoveContainer" containerID="5f87f4332bdc78c191aecf3ac6cf77c6090decc442e7c010e619e823409b827a"
Oct 29 23:08:58.416512 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.416490    5428 generic.go:356] "Generic (PLEG): container finished" podID="02f4bd43-6a70-471e-aa6c-459bdeaca665" containerID="b00eb59943c4265dc1de865f36186f3e3817c9d63b1fab108919301d376b40ae" exitCode=0
Oct 29 23:08:58.416624 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.416527    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z"
Oct 29 23:08:58.416624 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.416565    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z" event={"ID":"02f4bd43-6a70-471e-aa6c-459bdeaca665","Type":"ContainerDied","Data":"b00eb59943c4265dc1de865f36186f3e3817c9d63b1fab108919301d376b40ae"}
Oct 29 23:08:58.416624 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.416611    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z" event={"ID":"02f4bd43-6a70-471e-aa6c-459bdeaca665","Type":"ContainerDied","Data":"fd78681f7a087bf580c0732f98887b040383d929f3f1ba8f7cbf455a3f34f1f9"}
Oct 29 23:08:58.433874 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.433855    5428 scope.go:122] "RemoveContainer" containerID="5f87f4332bdc78c191aecf3ac6cf77c6090decc442e7c010e619e823409b827a"
Oct 29 23:08:58.434224 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:08:58.434200    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f87f4332bdc78c191aecf3ac6cf77c6090decc442e7c010e619e823409b827a\": container with ID starting with 5f87f4332bdc78c191aecf3ac6cf77c6090decc442e7c010e619e823409b827a not found: ID does not exist" containerID="5f87f4332bdc78c191aecf3ac6cf77c6090decc442e7c010e619e823409b827a"
Oct 29 23:08:58.434333 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.434232    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f87f4332bdc78c191aecf3ac6cf77c6090decc442e7c010e619e823409b827a"} err="failed to get container status \"5f87f4332bdc78c191aecf3ac6cf77c6090decc442e7c010e619e823409b827a\": rpc error: code = NotFound desc = could not find container \"5f87f4332bdc78c191aecf3ac6cf77c6090decc442e7c010e619e823409b827a\": container with ID starting with 5f87f4332bdc78c191aecf3ac6cf77c6090decc442e7c010e619e823409b827a not found: ID does not exist"
Oct 29 23:08:58.434333 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.434252    5428 scope.go:122] "RemoveContainer" containerID="b00eb59943c4265dc1de865f36186f3e3817c9d63b1fab108919301d376b40ae"
Oct 29 23:08:58.443741 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.443725    5428 scope.go:122] "RemoveContainer" containerID="b00eb59943c4265dc1de865f36186f3e3817c9d63b1fab108919301d376b40ae"
Oct 29 23:08:58.443994 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:08:58.443972    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00eb59943c4265dc1de865f36186f3e3817c9d63b1fab108919301d376b40ae\": container with ID starting with b00eb59943c4265dc1de865f36186f3e3817c9d63b1fab108919301d376b40ae not found: ID does not exist" containerID="b00eb59943c4265dc1de865f36186f3e3817c9d63b1fab108919301d376b40ae"
Oct 29 23:08:58.444054 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.444006    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00eb59943c4265dc1de865f36186f3e3817c9d63b1fab108919301d376b40ae"} err="failed to get container status \"b00eb59943c4265dc1de865f36186f3e3817c9d63b1fab108919301d376b40ae\": rpc error: code = NotFound desc = could not find container \"b00eb59943c4265dc1de865f36186f3e3817c9d63b1fab108919301d376b40ae\": container with ID starting with b00eb59943c4265dc1de865f36186f3e3817c9d63b1fab108919301d376b40ae not found: ID does not exist"
Oct 29 23:08:58.497981 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.497950    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa734e34-458a-4633-9caf-c1e1b97f5c6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa734e34-458a-4633-9caf-c1e1b97f5c6f-plugins-config-volume\") pod \"fa734e34-458a-4633-9caf-c1e1b97f5c6f\" (UID: \"fa734e34-458a-4633-9caf-c1e1b97f5c6f\") "
Oct 29 23:08:58.498189 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.498028    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f4bd43-6a70-471e-aa6c-459bdeaca665-kube-api-access-zhlc8\" (UniqueName: \"kubernetes.io/projected/02f4bd43-6a70-471e-aa6c-459bdeaca665-kube-api-access-zhlc8\") pod \"02f4bd43-6a70-471e-aa6c-459bdeaca665\" (UID: \"02f4bd43-6a70-471e-aa6c-459bdeaca665\") "
Oct 29 23:08:58.498189 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.498152    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa734e34-458a-4633-9caf-c1e1b97f5c6f-kube-api-access-rb8qh\" (UniqueName: \"kubernetes.io/projected/fa734e34-458a-4633-9caf-c1e1b97f5c6f-kube-api-access-rb8qh\") pod \"fa734e34-458a-4633-9caf-c1e1b97f5c6f\" (UID: \"fa734e34-458a-4633-9caf-c1e1b97f5c6f\") "
Oct 29 23:08:58.498356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.498339    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02f4bd43-6a70-471e-aa6c-459bdeaca665-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02f4bd43-6a70-471e-aa6c-459bdeaca665-plugins-config-volume\") pod \"02f4bd43-6a70-471e-aa6c-459bdeaca665\" (UID: \"02f4bd43-6a70-471e-aa6c-459bdeaca665\") "
Oct 29 23:08:58.498426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.498399    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa734e34-458a-4633-9caf-c1e1b97f5c6f-plugins-config-volume" pod "fa734e34-458a-4633-9caf-c1e1b97f5c6f" (UID: "fa734e34-458a-4633-9caf-c1e1b97f5c6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:58.498727 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.498704    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f4bd43-6a70-471e-aa6c-459bdeaca665-plugins-config-volume" pod "02f4bd43-6a70-471e-aa6c-459bdeaca665" (UID: "02f4bd43-6a70-471e-aa6c-459bdeaca665"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:58.498898 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.498881    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02f4bd43-6a70-471e-aa6c-459bdeaca665-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:58.498979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.498899    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa734e34-458a-4633-9caf-c1e1b97f5c6f-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:58.500348 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.500327    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa734e34-458a-4633-9caf-c1e1b97f5c6f-kube-api-access-rb8qh" pod "fa734e34-458a-4633-9caf-c1e1b97f5c6f" (UID: "fa734e34-458a-4633-9caf-c1e1b97f5c6f"). InnerVolumeSpecName "kube-api-access-rb8qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:58.500441 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.500361    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f4bd43-6a70-471e-aa6c-459bdeaca665-kube-api-access-zhlc8" pod "02f4bd43-6a70-471e-aa6c-459bdeaca665" (UID: "02f4bd43-6a70-471e-aa6c-459bdeaca665"). InnerVolumeSpecName "kube-api-access-zhlc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:58.599903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.599833    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhlc8\" (UniqueName: \"kubernetes.io/projected/02f4bd43-6a70-471e-aa6c-459bdeaca665-kube-api-access-zhlc8\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:58.599903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.599859    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb8qh\" (UniqueName: \"kubernetes.io/projected/fa734e34-458a-4633-9caf-c1e1b97f5c6f-kube-api-access-rb8qh\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:08:58.798164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.798127    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98"]
Oct 29 23:08:58.818652 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.818620    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c23236-6f96554868-crh98"]
Oct 29 23:08:58.853648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.853573    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z"]
Oct 29 23:08:58.857792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:58.857761    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c23236-b7d664fcf-ng57z"]
Oct 29 23:08:59.035745 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.035710    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"]
Oct 29 23:08:59.036374 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036360    5428 cpu_manager.go:414] "containerMap: removing container" podUID="76880e88-0196-4501-b499-52a0800e26dc" containerName="payload-processing"
Oct 29 23:08:59.036416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036377    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76880e88-0196-4501-b499-52a0800e26dc" containerName="payload-processing"
Oct 29 23:08:59.036416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036406    5428 cpu_manager.go:414] "containerMap: removing container" podUID="f8b78432-161a-418b-aeaf-a0580dc80b54" containerName="payload-pre-processing"
Oct 29 23:08:59.036416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036412    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b78432-161a-418b-aeaf-a0580dc80b54" containerName="payload-pre-processing"
Oct 29 23:08:59.036538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036420    5428 cpu_manager.go:414] "containerMap: removing container" podUID="2753c078-4e19-40c1-b4f0-c1591c6d64d4" containerName="maas-api"
Oct 29 23:08:59.036538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036425    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2753c078-4e19-40c1-b4f0-c1591c6d64d4" containerName="maas-api"
Oct 29 23:08:59.036538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036433    5428 cpu_manager.go:414] "containerMap: removing container" podUID="fa734e34-458a-4633-9caf-c1e1b97f5c6f" containerName="payload-processing"
Oct 29 23:08:59.036538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036439    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa734e34-458a-4633-9caf-c1e1b97f5c6f" containerName="payload-processing"
Oct 29 23:08:59.036538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036446    5428 cpu_manager.go:414] "containerMap: removing container" podUID="02f4bd43-6a70-471e-aa6c-459bdeaca665" containerName="payload-pre-processing"
Oct 29 23:08:59.036538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036451    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f4bd43-6a70-471e-aa6c-459bdeaca665" containerName="payload-pre-processing"
Oct 29 23:08:59.036538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036464    5428 cpu_manager.go:414] "containerMap: removing container" podUID="a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7" containerName="maas-api"
Oct 29 23:08:59.036538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036468    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7" containerName="maas-api"
Oct 29 23:08:59.036538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036476    5428 cpu_manager.go:414] "containerMap: removing container" podUID="3bca0741-fd7f-4183-b9f7-f5c827532f72" containerName="maas-api"
Oct 29 23:08:59.036538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036480    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bca0741-fd7f-4183-b9f7-f5c827532f72" containerName="maas-api"
Oct 29 23:08:59.036870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036565    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="76880e88-0196-4501-b499-52a0800e26dc" containerName="payload-processing"
Oct 29 23:08:59.036870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036577    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b78432-161a-418b-aeaf-a0580dc80b54" containerName="payload-pre-processing"
Oct 29 23:08:59.036870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036583    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="2753c078-4e19-40c1-b4f0-c1591c6d64d4" containerName="maas-api"
Oct 29 23:08:59.036870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036590    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa734e34-458a-4633-9caf-c1e1b97f5c6f" containerName="payload-processing"
Oct 29 23:08:59.036870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036596    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f4bd43-6a70-471e-aa6c-459bdeaca665" containerName="payload-pre-processing"
Oct 29 23:08:59.036870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036604    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f472b3-56b6-4901-a1a9-c4d0c07c6ad7" containerName="maas-api"
Oct 29 23:08:59.036870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.036611    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bca0741-fd7f-4183-b9f7-f5c827532f72" containerName="maas-api"
Oct 29 23:08:59.041570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.041549    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:08:59.065910 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.065877    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"]
Oct 29 23:08:59.106639 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.106553    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-maas-api-tls\") pod \"maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j\" (UID: \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:08:59.106639 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.106612    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-metrics-certs\") pod \"maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j\" (UID: \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:08:59.106918 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.106893    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j\" (UID: \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:08:59.106971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.106943    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w5m5\" (UniqueName: \"kubernetes.io/projected/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-kube-api-access-5w5m5\") pod \"maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j\" (UID: \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:08:59.111874 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.111838    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-31d798-serving-cert\""
Oct 29 23:08:59.143979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.143937    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:59.208111 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.208074    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j\" (UID: \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:08:59.208259 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.208119    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w5m5\" (UniqueName: \"kubernetes.io/projected/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-kube-api-access-5w5m5\") pod \"maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j\" (UID: \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:08:59.208259 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.208214    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-maas-api-tls\") pod \"maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j\" (UID: \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:08:59.208259 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.208252    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-metrics-certs\") pod \"maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j\" (UID: \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:08:59.208463 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:08:59.208427    5428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-serving-cert: secret "maas-api-e2e-ipp-b-31d798-serving-cert" not found
Oct 29 23:08:59.208588 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:08:59.208570    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-maas-api-tls podName:7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26 nodeName:}" failed. No retries permitted until 2026-10-29 23:08:59.708548597 +0000 UTC m=+1543.051497335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-maas-api-tls") pod "maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j" (UID: "7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26") : secret "maas-api-e2e-ipp-b-31d798-serving-cert" not found
Oct 29 23:08:59.208933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.208914    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j\" (UID: \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:08:59.210777 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.210751    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-metrics-certs\") pod \"maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j\" (UID: \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:08:59.231665 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.231631    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w5m5\" (UniqueName: \"kubernetes.io/projected/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-kube-api-access-5w5m5\") pod \"maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j\" (UID: \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:08:59.257836 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.257775    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:59.539455 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.539423    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:59.716194 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.716151    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-maas-api-tls\") pod \"maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j\" (UID: \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:08:59.718586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.718564    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-maas-api-tls\") pod \"maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j\" (UID: \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:08:59.967891 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:08:59.967854    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:09:00.282508 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:00.282467    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"]
Oct 29 23:09:00.284121 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:09:00.284088    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9a3e94_c03f_4ab1_8e78_e2a5aad2de26.slice/crio-34910c0ebf419d842faacf07cfa2a47c818199aa6b974353c17e1c671c4e435c WatchSource:0}: Error finding container 34910c0ebf419d842faacf07cfa2a47c818199aa6b974353c17e1c671c4e435c: Status 404 returned error can't find the container with id 34910c0ebf419d842faacf07cfa2a47c818199aa6b974353c17e1c671c4e435c
Oct 29 23:09:00.441411 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:00.441369    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j" event={"ID":"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26","Type":"ContainerStarted","Data":"34910c0ebf419d842faacf07cfa2a47c818199aa6b974353c17e1c671c4e435c"}
Oct 29 23:09:00.779328 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:00.779300    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02f4bd43-6a70-471e-aa6c-459bdeaca665" path="/var/lib/kubelet/pods/02f4bd43-6a70-471e-aa6c-459bdeaca665/volumes"
Oct 29 23:09:00.779772 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:00.779759    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa734e34-458a-4633-9caf-c1e1b97f5c6f" path="/var/lib/kubelet/pods/fa734e34-458a-4633-9caf-c1e1b97f5c6f/volumes"
Oct 29 23:09:01.020483 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.020458    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 23:09:01.140213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.140181    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-workload-socket\") pod \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") "
Oct 29 23:09:01.140397 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.140354    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istiod-ca-cert\") pod \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") "
Oct 29 23:09:01.140397 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.140390    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-envoy\") pod \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") "
Oct 29 23:09:01.140490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.140431    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a16c52-71c1-4da1-bc73-4e6fea6184c8-kube-api-access-p2hx5\" (UniqueName: \"kubernetes.io/projected/75a16c52-71c1-4da1-bc73-4e6fea6184c8-kube-api-access-p2hx5\") pod \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") "
Oct 29 23:09:01.140490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.140465    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-token\" (UniqueName: \"kubernetes.io/projected/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-token\") pod \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") "
Oct 29 23:09:01.140490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.140488    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-data\") pod \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") "
Oct 29 23:09:01.140632 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.140516    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-credential-socket\") pod \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") "
Oct 29 23:09:01.140632 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.140543    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-podinfo\") pod \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") "
Oct 29 23:09:01.140731 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.140685    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-workload-certs\") pod \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\" (UID: \"75a16c52-71c1-4da1-bc73-4e6fea6184c8\") "
Oct 29 23:09:01.141113 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.140763    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istiod-ca-cert" pod "75a16c52-71c1-4da1-bc73-4e6fea6184c8" (UID: "75a16c52-71c1-4da1-bc73-4e6fea6184c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:01.141113 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.141015    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-workload-socket" pod "75a16c52-71c1-4da1-bc73-4e6fea6184c8" (UID: "75a16c52-71c1-4da1-bc73-4e6fea6184c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:01.141113 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.141032    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-data" pod "75a16c52-71c1-4da1-bc73-4e6fea6184c8" (UID: "75a16c52-71c1-4da1-bc73-4e6fea6184c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:01.141453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.141430    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-credential-socket" pod "75a16c52-71c1-4da1-bc73-4e6fea6184c8" (UID: "75a16c52-71c1-4da1-bc73-4e6fea6184c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:01.141599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.141341    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:01.141599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.141545    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:01.141599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.141562    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:01.141599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.141566    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-workload-certs" pod "75a16c52-71c1-4da1-bc73-4e6fea6184c8" (UID: "75a16c52-71c1-4da1-bc73-4e6fea6184c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:01.143042 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.143021    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-podinfo" pod "75a16c52-71c1-4da1-bc73-4e6fea6184c8" (UID: "75a16c52-71c1-4da1-bc73-4e6fea6184c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:09:01.143124 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.143019    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-envoy" pod "75a16c52-71c1-4da1-bc73-4e6fea6184c8" (UID: "75a16c52-71c1-4da1-bc73-4e6fea6184c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:09:01.143233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.143218    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a16c52-71c1-4da1-bc73-4e6fea6184c8-kube-api-access-p2hx5" pod "75a16c52-71c1-4da1-bc73-4e6fea6184c8" (UID: "75a16c52-71c1-4da1-bc73-4e6fea6184c8"). InnerVolumeSpecName "kube-api-access-p2hx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:01.143444 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.143420    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-token" pod "75a16c52-71c1-4da1-bc73-4e6fea6184c8" (UID: "75a16c52-71c1-4da1-bc73-4e6fea6184c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:01.191564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.191493    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:01.242655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.242619    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:01.242655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.242649    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:01.242655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.242660    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2hx5\" (UniqueName: \"kubernetes.io/projected/75a16c52-71c1-4da1-bc73-4e6fea6184c8-kube-api-access-p2hx5\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:01.242915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.242669    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:01.242915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.242679    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a16c52-71c1-4da1-bc73-4e6fea6184c8-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:01.242915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.242687    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a16c52-71c1-4da1-bc73-4e6fea6184c8-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:01.310382 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.310105    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:01.451179 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.451094    5428 generic.go:356] "Generic (PLEG): container finished" podID="75a16c52-71c1-4da1-bc73-4e6fea6184c8" containerID="f5af76081a2c09f4a2b14c5fda368876ea76b3b70c9098a3bd55340fa656711e" exitCode=0
Oct 29 23:09:01.451179 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.451145    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"
Oct 29 23:09:01.451992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.451184    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x" event={"ID":"75a16c52-71c1-4da1-bc73-4e6fea6184c8","Type":"ContainerDied","Data":"f5af76081a2c09f4a2b14c5fda368876ea76b3b70c9098a3bd55340fa656711e"}
Oct 29 23:09:01.451992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.451244    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x" event={"ID":"75a16c52-71c1-4da1-bc73-4e6fea6184c8","Type":"ContainerDied","Data":"88040f1faadc2f2e3641a81aa8a26c9d58cbe7cc56977289ba4733d333a9d1ea"}
Oct 29 23:09:01.451992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.451277    5428 scope.go:122] "RemoveContainer" containerID="f5af76081a2c09f4a2b14c5fda368876ea76b3b70c9098a3bd55340fa656711e"
Oct 29 23:09:01.453223 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.453197    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j" event={"ID":"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26","Type":"ContainerStarted","Data":"6a3339763001bdcada2bb9e561cf5d0853de8352b7deaa979ff59dd46996355f"}
Oct 29 23:09:01.453356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.453270    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:09:01.464213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.464181    5428 scope.go:122] "RemoveContainer" containerID="f5af76081a2c09f4a2b14c5fda368876ea76b3b70c9098a3bd55340fa656711e"
Oct 29 23:09:01.464494 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:09:01.464475    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5af76081a2c09f4a2b14c5fda368876ea76b3b70c9098a3bd55340fa656711e\": container with ID starting with f5af76081a2c09f4a2b14c5fda368876ea76b3b70c9098a3bd55340fa656711e not found: ID does not exist" containerID="f5af76081a2c09f4a2b14c5fda368876ea76b3b70c9098a3bd55340fa656711e"
Oct 29 23:09:01.464579 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.464507    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5af76081a2c09f4a2b14c5fda368876ea76b3b70c9098a3bd55340fa656711e"} err="failed to get container status \"f5af76081a2c09f4a2b14c5fda368876ea76b3b70c9098a3bd55340fa656711e\": rpc error: code = NotFound desc = could not find container \"f5af76081a2c09f4a2b14c5fda368876ea76b3b70c9098a3bd55340fa656711e\": container with ID starting with f5af76081a2c09f4a2b14c5fda368876ea76b3b70c9098a3bd55340fa656711e not found: ID does not exist"
Oct 29 23:09:01.517757 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.517708    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j" podStartSLOduration=2.149855143 podStartE2EDuration="2.517692493s" totalImagesPullingTime="367.83735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:01.515058789 +0000 UTC m=+1544.858007535" watchObservedRunningTime="2026-10-29 23:09:01.517692493 +0000 UTC m=+1544.860641266"
Oct 29 23:09:01.553992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.553964    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"]
Oct 29 23:09:01.560191 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:01.560162    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-8vp4x"]
Oct 29 23:09:02.442543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:02.442513    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:02.779262 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:02.779179    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a16c52-71c1-4da1-bc73-4e6fea6184c8" path="/var/lib/kubelet/pods/75a16c52-71c1-4da1-bc73-4e6fea6184c8/volumes"
Oct 29 23:09:02.994398 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:02.994352    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:03.119632 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:03.119600    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:03.653066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:03.653027    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:04.376331 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:04.376295    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:04.488981 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:04.488944    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:04.828061 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:04.827936    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:05.331278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:05.331248    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:06.370212 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:06.370175    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:06.573147 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:06.573106    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:06.695580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:06.695469    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:06.863110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:06.863079    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:07.372623 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:07.372585    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:07.468150 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:07.468124    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:09:07.560132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:07.560099    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:08.233818 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.233768    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:08.359484 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.359451    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:08.391516 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.391482    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l"]
Oct 29 23:09:08.392151 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.392136    5428 cpu_manager.go:414] "containerMap: removing container" podUID="75a16c52-71c1-4da1-bc73-4e6fea6184c8" containerName="istio-proxy"
Oct 29 23:09:08.392214 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.392152    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a16c52-71c1-4da1-bc73-4e6fea6184c8" containerName="istio-proxy"
Oct 29 23:09:08.392294 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.392285    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a16c52-71c1-4da1-bc73-4e6fea6184c8" containerName="istio-proxy"
Oct 29 23:09:08.395996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.395981    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l"
Oct 29 23:09:08.415092 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.415062    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l"]
Oct 29 23:09:08.423067 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.423036    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-85c702-dockercfg-fsfnl\""
Oct 29 23:09:08.423221 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.423095    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-85c702\""
Oct 29 23:09:08.453404 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.453362    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181b8c86-e1c1-4208-9c6b-8a249d58c2a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l\" (UID: \"181b8c86-e1c1-4208-9c6b-8a249d58c2a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l"
Oct 29 23:09:08.453643 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.453472    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bswhv\" (UniqueName: \"kubernetes.io/projected/181b8c86-e1c1-4208-9c6b-8a249d58c2a8-kube-api-access-bswhv\") pod \"payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l\" (UID: \"181b8c86-e1c1-4208-9c6b-8a249d58c2a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l"
Oct 29 23:09:08.497242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.497161    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx"]
Oct 29 23:09:08.502701 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.502671    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx"
Oct 29 23:09:08.532862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.526857    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx"]
Oct 29 23:09:08.556114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.556066    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38d4301-782b-4d87-a7f3-3b6036648288-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx\" (UID: \"b38d4301-782b-4d87-a7f3-3b6036648288\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx"
Oct 29 23:09:08.556319 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.556227    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181b8c86-e1c1-4208-9c6b-8a249d58c2a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l\" (UID: \"181b8c86-e1c1-4208-9c6b-8a249d58c2a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l"
Oct 29 23:09:08.556319 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.556297    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bswhv\" (UniqueName: \"kubernetes.io/projected/181b8c86-e1c1-4208-9c6b-8a249d58c2a8-kube-api-access-bswhv\") pod \"payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l\" (UID: \"181b8c86-e1c1-4208-9c6b-8a249d58c2a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l"
Oct 29 23:09:08.556494 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.556472    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79s7z\" (UniqueName: \"kubernetes.io/projected/b38d4301-782b-4d87-a7f3-3b6036648288-kube-api-access-79s7z\") pod \"payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx\" (UID: \"b38d4301-782b-4d87-a7f3-3b6036648288\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx"
Oct 29 23:09:08.558176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.557494    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181b8c86-e1c1-4208-9c6b-8a249d58c2a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l\" (UID: \"181b8c86-e1c1-4208-9c6b-8a249d58c2a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l"
Oct 29 23:09:08.585668 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.585450    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bswhv\" (UniqueName: \"kubernetes.io/projected/181b8c86-e1c1-4208-9c6b-8a249d58c2a8-kube-api-access-bswhv\") pod \"payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l\" (UID: \"181b8c86-e1c1-4208-9c6b-8a249d58c2a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l"
Oct 29 23:09:08.657914 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.657870    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79s7z\" (UniqueName: \"kubernetes.io/projected/b38d4301-782b-4d87-a7f3-3b6036648288-kube-api-access-79s7z\") pod \"payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx\" (UID: \"b38d4301-782b-4d87-a7f3-3b6036648288\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx"
Oct 29 23:09:08.658111 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.658006    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38d4301-782b-4d87-a7f3-3b6036648288-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx\" (UID: \"b38d4301-782b-4d87-a7f3-3b6036648288\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx"
Oct 29 23:09:08.659135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.659064    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38d4301-782b-4d87-a7f3-3b6036648288-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx\" (UID: \"b38d4301-782b-4d87-a7f3-3b6036648288\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx"
Oct 29 23:09:08.681501 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.681460    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79s7z\" (UniqueName: \"kubernetes.io/projected/b38d4301-782b-4d87-a7f3-3b6036648288-kube-api-access-79s7z\") pod \"payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx\" (UID: \"b38d4301-782b-4d87-a7f3-3b6036648288\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx"
Oct 29 23:09:08.711048 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.711016    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l"
Oct 29 23:09:08.835049 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:08.835016    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx"
Oct 29 23:09:09.037294 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:09.037263    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l"]
Oct 29 23:09:09.039084 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:09:09.039054    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181b8c86_e1c1_4208_9c6b_8a249d58c2a8.slice/crio-d5e792de4aad3822008d505dab31d36ab18cda04f330bdba38e2943c84b2dd2f WatchSource:0}: Error finding container d5e792de4aad3822008d505dab31d36ab18cda04f330bdba38e2943c84b2dd2f: Status 404 returned error can't find the container with id d5e792de4aad3822008d505dab31d36ab18cda04f330bdba38e2943c84b2dd2f
Oct 29 23:09:09.159174 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:09.159149    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx"]
Oct 29 23:09:09.522522 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:09.522487    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l" event={"ID":"181b8c86-e1c1-4208-9c6b-8a249d58c2a8","Type":"ContainerStarted","Data":"97b0f298efef23c17e595339a6bc3e9333f7bd88c73544b7c7e252c24451ab54"}
Oct 29 23:09:09.522939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:09.522539    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l" event={"ID":"181b8c86-e1c1-4208-9c6b-8a249d58c2a8","Type":"ContainerStarted","Data":"d5e792de4aad3822008d505dab31d36ab18cda04f330bdba38e2943c84b2dd2f"}
Oct 29 23:09:09.522939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:09.522600    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l"
Oct 29 23:09:09.524256 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:09.524231    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx" event={"ID":"b38d4301-782b-4d87-a7f3-3b6036648288","Type":"ContainerStarted","Data":"3c73136f273803a9aa7af3ee5b786a43604500e18798b45b765d5314eb5678c6"}
Oct 29 23:09:09.524359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:09.524275    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx"
Oct 29 23:09:09.524359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:09.524290    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx" event={"ID":"b38d4301-782b-4d87-a7f3-3b6036648288","Type":"ContainerStarted","Data":"1d306c87b37e413ec8512f6a4afc7a6efd651940c610b77f2c3f090619888633"}
Oct 29 23:09:09.565744 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:09.565696    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l" podStartSLOduration=1.565683122 podStartE2EDuration="1.565683122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:09.562462464 +0000 UTC m=+1552.905411212" watchObservedRunningTime="2026-10-29 23:09:09.565683122 +0000 UTC m=+1552.908631866"
Oct 29 23:09:09.606129 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:09.606075    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx" podStartSLOduration=1.6060613529999999 podStartE2EDuration="1.606061353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:09.603142022 +0000 UTC m=+1552.946090790" watchObservedRunningTime="2026-10-29 23:09:09.606061353 +0000 UTC m=+1552.949010097"
Oct 29 23:09:10.343618 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:10.343579    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:10.790424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:10.785504    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:10.922467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:10.922434    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:12.440572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:12.440540    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:12.574181 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:12.574142    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:12.622905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:12.622873    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27"]
Oct 29 23:09:12.627828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:12.627777    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27"
Oct 29 23:09:12.647517 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:12.647452    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27"]
Oct 29 23:09:12.676180 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:12.676143    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"]
Oct 29 23:09:12.676536 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:12.676416    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq" podUID="1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5" containerName="maas-api" containerID="cri-o://2ad2ba0343adeadc910f33b97dfdbeae649b36144fb1e37c12ee6fe0af494035" gracePeriod=30
Oct 29 23:09:12.715066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:12.714194    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f2c8d3-dockercfg-mktwv\""
Oct 29 23:09:12.715066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:12.714287    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f2c8d3\""
Oct 29 23:09:12.728813 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:12.720716    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96hgj\" (UniqueName: \"kubernetes.io/projected/9f7ba3a0-58c9-4bfb-a0d7-cf7786563277-kube-api-access-96hgj\") pod \"payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27\" (UID: \"9f7ba3a0-58c9-4bfb-a0d7-cf7786563277\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27"
Oct 29 23:09:12.728813 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:12.720876    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7ba3a0-58c9-4bfb-a0d7-cf7786563277-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27\" (UID: \"9f7ba3a0-58c9-4bfb-a0d7-cf7786563277\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27"
Oct 29 23:09:12.827040 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:12.826098    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96hgj\" (UniqueName: \"kubernetes.io/projected/9f7ba3a0-58c9-4bfb-a0d7-cf7786563277-kube-api-access-96hgj\") pod \"payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27\" (UID: \"9f7ba3a0-58c9-4bfb-a0d7-cf7786563277\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27"
Oct 29 23:09:12.827285 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:12.827171    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7ba3a0-58c9-4bfb-a0d7-cf7786563277-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27\" (UID: \"9f7ba3a0-58c9-4bfb-a0d7-cf7786563277\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27"
Oct 29 23:09:12.828106 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:12.828068    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:12.828859 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:12.828823    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7ba3a0-58c9-4bfb-a0d7-cf7786563277-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27\" (UID: \"9f7ba3a0-58c9-4bfb-a0d7-cf7786563277\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27"
Oct 29 23:09:12.871120 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:12.871086    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96hgj\" (UniqueName: \"kubernetes.io/projected/9f7ba3a0-58c9-4bfb-a0d7-cf7786563277-kube-api-access-96hgj\") pod \"payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27\" (UID: \"9f7ba3a0-58c9-4bfb-a0d7-cf7786563277\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27"
Oct 29 23:09:12.942762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:12.942720    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27"
Oct 29 23:09:13.033461 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.033074    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:09:13.142187 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.142145    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-metrics-certs\") pod \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\" (UID: \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\") "
Oct 29 23:09:13.142350 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.142274    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-maas-api-tls\") pod \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\" (UID: \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\") "
Oct 29 23:09:13.142350 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.142321    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-kube-api-access-qqcdk\" (UniqueName: \"kubernetes.io/projected/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-kube-api-access-qqcdk\") pod \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\" (UID: \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\") "
Oct 29 23:09:13.142350 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.142346    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-openshift-service-ca\") pod \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\" (UID: \"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5\") "
Oct 29 23:09:13.142837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.142783    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-openshift-service-ca" pod "1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5" (UID: "1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:13.144315 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.144293    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-metrics-certs" pod "1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5" (UID: "1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:09:13.144387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.144367    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-kube-api-access-qqcdk" pod "1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5" (UID: "1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5"). InnerVolumeSpecName "kube-api-access-qqcdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:13.144429 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.144372    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-maas-api-tls" pod "1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5" (UID: "1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:09:13.244619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.244545    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:13.244619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.244574    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqcdk\" (UniqueName: \"kubernetes.io/projected/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-kube-api-access-qqcdk\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:13.244619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.244586    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:13.244619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.244595    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:13.275454 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.275409    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27"]
Oct 29 23:09:13.277259 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:09:13.277222    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7ba3a0_58c9_4bfb_a0d7_cf7786563277.slice/crio-9490dbd28c50cb88c669e29298ac8e76f172f734ed3ae5e5dcc913e8a21d7248 WatchSource:0}: Error finding container 9490dbd28c50cb88c669e29298ac8e76f172f734ed3ae5e5dcc913e8a21d7248: Status 404 returned error can't find the container with id 9490dbd28c50cb88c669e29298ac8e76f172f734ed3ae5e5dcc913e8a21d7248
Oct 29 23:09:13.556625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.556518    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27" event={"ID":"9f7ba3a0-58c9-4bfb-a0d7-cf7786563277","Type":"ContainerStarted","Data":"3e4e4f5eb190572d2e17214968f776d2756779115e8f52e26de45d9aad9f7a90"}
Oct 29 23:09:13.556625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.556582    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27" event={"ID":"9f7ba3a0-58c9-4bfb-a0d7-cf7786563277","Type":"ContainerStarted","Data":"9490dbd28c50cb88c669e29298ac8e76f172f734ed3ae5e5dcc913e8a21d7248"}
Oct 29 23:09:13.556625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.556612    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27"
Oct 29 23:09:13.558159 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.558125    5428 generic.go:356] "Generic (PLEG): container finished" podID="1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5" containerID="2ad2ba0343adeadc910f33b97dfdbeae649b36144fb1e37c12ee6fe0af494035" exitCode=0
Oct 29 23:09:13.558522 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.558194    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"
Oct 29 23:09:13.558522 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.558218    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq" event={"ID":"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5","Type":"ContainerDied","Data":"2ad2ba0343adeadc910f33b97dfdbeae649b36144fb1e37c12ee6fe0af494035"}
Oct 29 23:09:13.558522 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.558292    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq" event={"ID":"1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5","Type":"ContainerDied","Data":"2323d392861da1fa18666e429eae49afa396f7715b93f1e4bf5d5517f370702b"}
Oct 29 23:09:13.558522 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.558335    5428 scope.go:122] "RemoveContainer" containerID="2ad2ba0343adeadc910f33b97dfdbeae649b36144fb1e37c12ee6fe0af494035"
Oct 29 23:09:13.570447 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.570426    5428 scope.go:122] "RemoveContainer" containerID="2ad2ba0343adeadc910f33b97dfdbeae649b36144fb1e37c12ee6fe0af494035"
Oct 29 23:09:13.570741 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:09:13.570717    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad2ba0343adeadc910f33b97dfdbeae649b36144fb1e37c12ee6fe0af494035\": container with ID starting with 2ad2ba0343adeadc910f33b97dfdbeae649b36144fb1e37c12ee6fe0af494035 not found: ID does not exist" containerID="2ad2ba0343adeadc910f33b97dfdbeae649b36144fb1e37c12ee6fe0af494035"
Oct 29 23:09:13.570894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.570751    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad2ba0343adeadc910f33b97dfdbeae649b36144fb1e37c12ee6fe0af494035"} err="failed to get container status \"2ad2ba0343adeadc910f33b97dfdbeae649b36144fb1e37c12ee6fe0af494035\": rpc error: code = NotFound desc = could not find container \"2ad2ba0343adeadc910f33b97dfdbeae649b36144fb1e37c12ee6fe0af494035\": container with ID starting with 2ad2ba0343adeadc910f33b97dfdbeae649b36144fb1e37c12ee6fe0af494035 not found: ID does not exist"
Oct 29 23:09:13.600964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.600914    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27" podStartSLOduration=1.600899028 podStartE2EDuration="1.600899028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:13.595918842 +0000 UTC m=+1556.938867615" watchObservedRunningTime="2026-10-29 23:09:13.600899028 +0000 UTC m=+1556.943847789"
Oct 29 23:09:13.629938 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.629904    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"]
Oct 29 23:09:13.630173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.630140    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28" podUID="e77d6b76-753a-46f4-aab0-c35757e5f4bc" containerName="maas-api" containerID="cri-o://2ecb8222f2ceead08cba8c775d92a045df76dde12121f9757b513b7c2eac8d29" gracePeriod=30
Oct 29 23:09:13.639651 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.639585    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"]
Oct 29 23:09:13.653263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.647976    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd000a-77cd6d7f5f-gmpvq"]
Oct 29 23:09:13.718994 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.718960    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:13.938717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:13.938686    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:09:14.006335 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.006304    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:14.074479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.074399    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77d6b76-753a-46f4-aab0-c35757e5f4bc-kube-api-access-49vsq\" (UniqueName: \"kubernetes.io/projected/e77d6b76-753a-46f4-aab0-c35757e5f4bc-kube-api-access-49vsq\") pod \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\" (UID: \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\") "
Oct 29 23:09:14.074479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.074458    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77d6b76-753a-46f4-aab0-c35757e5f4bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77d6b76-753a-46f4-aab0-c35757e5f4bc-openshift-service-ca\") pod \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\" (UID: \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\") "
Oct 29 23:09:14.074767 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.074533    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e77d6b76-753a-46f4-aab0-c35757e5f4bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e77d6b76-753a-46f4-aab0-c35757e5f4bc-metrics-certs\") pod \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\" (UID: \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\") "
Oct 29 23:09:14.074767 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.074620    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e77d6b76-753a-46f4-aab0-c35757e5f4bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e77d6b76-753a-46f4-aab0-c35757e5f4bc-maas-api-tls\") pod \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\" (UID: \"e77d6b76-753a-46f4-aab0-c35757e5f4bc\") "
Oct 29 23:09:14.074895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.074842    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77d6b76-753a-46f4-aab0-c35757e5f4bc-openshift-service-ca" pod "e77d6b76-753a-46f4-aab0-c35757e5f4bc" (UID: "e77d6b76-753a-46f4-aab0-c35757e5f4bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:14.075286 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.075255    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77d6b76-753a-46f4-aab0-c35757e5f4bc-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:14.076842 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.076792    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e77d6b76-753a-46f4-aab0-c35757e5f4bc-metrics-certs" pod "e77d6b76-753a-46f4-aab0-c35757e5f4bc" (UID: "e77d6b76-753a-46f4-aab0-c35757e5f4bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:09:14.076989 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.076971    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77d6b76-753a-46f4-aab0-c35757e5f4bc-kube-api-access-49vsq" pod "e77d6b76-753a-46f4-aab0-c35757e5f4bc" (UID: "e77d6b76-753a-46f4-aab0-c35757e5f4bc"). InnerVolumeSpecName "kube-api-access-49vsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:14.077088 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.077072    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e77d6b76-753a-46f4-aab0-c35757e5f4bc-maas-api-tls" pod "e77d6b76-753a-46f4-aab0-c35757e5f4bc" (UID: "e77d6b76-753a-46f4-aab0-c35757e5f4bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:09:14.158241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.158201    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:14.176540 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.176515    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49vsq\" (UniqueName: \"kubernetes.io/projected/e77d6b76-753a-46f4-aab0-c35757e5f4bc-kube-api-access-49vsq\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:14.176540 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.176540    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e77d6b76-753a-46f4-aab0-c35757e5f4bc-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:14.176698 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.176551    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e77d6b76-753a-46f4-aab0-c35757e5f4bc-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:14.571171 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.571137    5428 generic.go:356] "Generic (PLEG): container finished" podID="e77d6b76-753a-46f4-aab0-c35757e5f4bc" containerID="2ecb8222f2ceead08cba8c775d92a045df76dde12121f9757b513b7c2eac8d29" exitCode=0
Oct 29 23:09:14.571586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.571187    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"
Oct 29 23:09:14.571586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.571220    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28" event={"ID":"e77d6b76-753a-46f4-aab0-c35757e5f4bc","Type":"ContainerDied","Data":"2ecb8222f2ceead08cba8c775d92a045df76dde12121f9757b513b7c2eac8d29"}
Oct 29 23:09:14.571586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.571279    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28" event={"ID":"e77d6b76-753a-46f4-aab0-c35757e5f4bc","Type":"ContainerDied","Data":"0cb71508926f7e787211e6af65d6139029fe2d21031c0c6f418347393469c6f0"}
Oct 29 23:09:14.571586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.571310    5428 scope.go:122] "RemoveContainer" containerID="2ecb8222f2ceead08cba8c775d92a045df76dde12121f9757b513b7c2eac8d29"
Oct 29 23:09:14.582931 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.582908    5428 scope.go:122] "RemoveContainer" containerID="2ecb8222f2ceead08cba8c775d92a045df76dde12121f9757b513b7c2eac8d29"
Oct 29 23:09:14.583261 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:09:14.583236    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ecb8222f2ceead08cba8c775d92a045df76dde12121f9757b513b7c2eac8d29\": container with ID starting with 2ecb8222f2ceead08cba8c775d92a045df76dde12121f9757b513b7c2eac8d29 not found: ID does not exist" containerID="2ecb8222f2ceead08cba8c775d92a045df76dde12121f9757b513b7c2eac8d29"
Oct 29 23:09:14.583348 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.583273    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ecb8222f2ceead08cba8c775d92a045df76dde12121f9757b513b7c2eac8d29"} err="failed to get container status \"2ecb8222f2ceead08cba8c775d92a045df76dde12121f9757b513b7c2eac8d29\": rpc error: code = NotFound desc = could not find container \"2ecb8222f2ceead08cba8c775d92a045df76dde12121f9757b513b7c2eac8d29\": container with ID starting with 2ecb8222f2ceead08cba8c775d92a045df76dde12121f9757b513b7c2eac8d29 not found: ID does not exist"
Oct 29 23:09:14.625225 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.625183    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"]
Oct 29 23:09:14.639070 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.639008    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c23236-994b96866-4kt28"]
Oct 29 23:09:14.664076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.664036    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:14.779605 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.779572    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5" path="/var/lib/kubelet/pods/1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5/volumes"
Oct 29 23:09:14.780076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.780061    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e77d6b76-753a-46f4-aab0-c35757e5f4bc" path="/var/lib/kubelet/pods/e77d6b76-753a-46f4-aab0-c35757e5f4bc/volumes"
Oct 29 23:09:14.810399 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.810372    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:14.965082 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:14.965049    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:15.285995 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:15.285899    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:15.561374 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:15.561289    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:16.681919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:16.681881    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:16.794081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:16.794034    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:17.242528 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:17.242484    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:17.372644 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:09:17.372603    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:09:19.487096 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:19.487061    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:20.534787 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:20.534759    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l"
Oct 29 23:09:20.535223 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:20.534853    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx"
Oct 29 23:09:21.170124 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:21.170091    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:21.285641 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:21.285601    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:21.577366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:21.577288    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:22.148969 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:22.148935    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:22.265199 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:22.265165    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:22.510067 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:22.509985    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:23.324191 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:23.324155    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:24.574987 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:24.574950    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27"
Oct 29 23:09:25.300013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:25.299957    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:26.848659 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:26.848623    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:27.063461 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:27.063427    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:27.642556 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:27.642518    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:28.855879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:28.855833    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:30.961123 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:30.961089    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:32.678359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:32.678303    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:34.343606 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:34.343565    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:34.461239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:34.461202    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:34.578056 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:34.578014    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:34.824930 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:34.824898    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg"]
Oct 29 23:09:34.929519 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:34.929481    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:35.050683 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:35.050647    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:36.397700 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:36.397650    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:38.307301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:38.307262    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:39.825516 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:39.825482    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg" podUID="85bd1ee8-788b-492f-a1ed-e09be149f040" containerName="payload-pre-processing" containerID="cri-o://760b70ebd3b42e6bcd7472e437e5ec4db80bc4dc056f4b35ec4a3e49dd14553e" gracePeriod=25
Oct 29 23:09:40.092638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.092572    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg"
Oct 29 23:09:40.254697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.254668    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85bd1ee8-788b-492f-a1ed-e09be149f040-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85bd1ee8-788b-492f-a1ed-e09be149f040-plugins-config-volume\") pod \"85bd1ee8-788b-492f-a1ed-e09be149f040\" (UID: \"85bd1ee8-788b-492f-a1ed-e09be149f040\") "
Oct 29 23:09:40.255038 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.255018    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85bd1ee8-788b-492f-a1ed-e09be149f040-kube-api-access-mg7lp\" (UniqueName: \"kubernetes.io/projected/85bd1ee8-788b-492f-a1ed-e09be149f040-kube-api-access-mg7lp\") pod \"85bd1ee8-788b-492f-a1ed-e09be149f040\" (UID: \"85bd1ee8-788b-492f-a1ed-e09be149f040\") "
Oct 29 23:09:40.255160 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.255017    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85bd1ee8-788b-492f-a1ed-e09be149f040-plugins-config-volume" pod "85bd1ee8-788b-492f-a1ed-e09be149f040" (UID: "85bd1ee8-788b-492f-a1ed-e09be149f040"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:40.255507 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.255488    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85bd1ee8-788b-492f-a1ed-e09be149f040-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:40.257047 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.257024    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85bd1ee8-788b-492f-a1ed-e09be149f040-kube-api-access-mg7lp" pod "85bd1ee8-788b-492f-a1ed-e09be149f040" (UID: "85bd1ee8-788b-492f-a1ed-e09be149f040"). InnerVolumeSpecName "kube-api-access-mg7lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:40.356960 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.356890    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg7lp\" (UniqueName: \"kubernetes.io/projected/85bd1ee8-788b-492f-a1ed-e09be149f040-kube-api-access-mg7lp\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:09:40.673815 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.673766    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"]
Oct 29 23:09:40.674989 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.674962    5428 cpu_manager.go:414] "containerMap: removing container" podUID="85bd1ee8-788b-492f-a1ed-e09be149f040" containerName="payload-pre-processing"
Oct 29 23:09:40.674989 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.674984    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85bd1ee8-788b-492f-a1ed-e09be149f040" containerName="payload-pre-processing"
Oct 29 23:09:40.675176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.675008    5428 cpu_manager.go:414] "containerMap: removing container" podUID="1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5" containerName="maas-api"
Oct 29 23:09:40.675176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.675014    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5" containerName="maas-api"
Oct 29 23:09:40.675176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.675045    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e77d6b76-753a-46f4-aab0-c35757e5f4bc" containerName="maas-api"
Oct 29 23:09:40.675176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.675055    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77d6b76-753a-46f4-aab0-c35757e5f4bc" containerName="maas-api"
Oct 29 23:09:40.675360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.675189    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77d6b76-753a-46f4-aab0-c35757e5f4bc" containerName="maas-api"
Oct 29 23:09:40.675360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.675206    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="85bd1ee8-788b-492f-a1ed-e09be149f040" containerName="payload-pre-processing"
Oct 29 23:09:40.675360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.675215    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb71ae1-c2d7-4d7a-8ebf-72c97bbf92a5" containerName="maas-api"
Oct 29 23:09:40.679069 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.679044    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"
Oct 29 23:09:40.705067 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.702775    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"]
Oct 29 23:09:40.706109 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.706079    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-f2c8d3-serving-cert\""
Oct 29 23:09:40.765831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.765766    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:40.835173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.835138    5428 generic.go:356] "Generic (PLEG): container finished" podID="85bd1ee8-788b-492f-a1ed-e09be149f040" containerID="760b70ebd3b42e6bcd7472e437e5ec4db80bc4dc056f4b35ec4a3e49dd14553e" exitCode=0
Oct 29 23:09:40.835661 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.835203    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg"
Oct 29 23:09:40.835661 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.835220    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg" event={"ID":"85bd1ee8-788b-492f-a1ed-e09be149f040","Type":"ContainerDied","Data":"760b70ebd3b42e6bcd7472e437e5ec4db80bc4dc056f4b35ec4a3e49dd14553e"}
Oct 29 23:09:40.835661 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.835273    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg" event={"ID":"85bd1ee8-788b-492f-a1ed-e09be149f040","Type":"ContainerDied","Data":"be45d271f8939d0d8ee1ab3d895bda4dfa641c56c8993bc3ced2c94baac4b317"}
Oct 29 23:09:40.835661 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.835295    5428 scope.go:122] "RemoveContainer" containerID="760b70ebd3b42e6bcd7472e437e5ec4db80bc4dc056f4b35ec4a3e49dd14553e"
Oct 29 23:09:40.845247 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.845227    5428 scope.go:122] "RemoveContainer" containerID="760b70ebd3b42e6bcd7472e437e5ec4db80bc4dc056f4b35ec4a3e49dd14553e"
Oct 29 23:09:40.845521 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:09:40.845500    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"760b70ebd3b42e6bcd7472e437e5ec4db80bc4dc056f4b35ec4a3e49dd14553e\": container with ID starting with 760b70ebd3b42e6bcd7472e437e5ec4db80bc4dc056f4b35ec4a3e49dd14553e not found: ID does not exist" containerID="760b70ebd3b42e6bcd7472e437e5ec4db80bc4dc056f4b35ec4a3e49dd14553e"
Oct 29 23:09:40.845567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.845527    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"760b70ebd3b42e6bcd7472e437e5ec4db80bc4dc056f4b35ec4a3e49dd14553e"} err="failed to get container status \"760b70ebd3b42e6bcd7472e437e5ec4db80bc4dc056f4b35ec4a3e49dd14553e\": rpc error: code = NotFound desc = could not find container \"760b70ebd3b42e6bcd7472e437e5ec4db80bc4dc056f4b35ec4a3e49dd14553e\": container with ID starting with 760b70ebd3b42e6bcd7472e437e5ec4db80bc4dc056f4b35ec4a3e49dd14553e not found: ID does not exist"
Oct 29 23:09:40.863853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.863826    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412ce42d-92e1-4a65-9378-e78592256758-metrics-certs\") pod \"maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv\" (UID: \"412ce42d-92e1-4a65-9378-e78592256758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"
Oct 29 23:09:40.864056 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.864040    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/412ce42d-92e1-4a65-9378-e78592256758-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv\" (UID: \"412ce42d-92e1-4a65-9378-e78592256758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"
Oct 29 23:09:40.864158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.864140    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brd5x\" (UniqueName: \"kubernetes.io/projected/412ce42d-92e1-4a65-9378-e78592256758-kube-api-access-brd5x\") pod \"maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv\" (UID: \"412ce42d-92e1-4a65-9378-e78592256758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"
Oct 29 23:09:40.864204 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.864181    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/412ce42d-92e1-4a65-9378-e78592256758-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv\" (UID: \"412ce42d-92e1-4a65-9378-e78592256758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"
Oct 29 23:09:40.878521 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.878494    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:40.887563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.887535    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg"]
Oct 29 23:09:40.892905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.892881    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-31d798-6496cf4c-tj9mg"]
Oct 29 23:09:40.965201 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.965119    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/412ce42d-92e1-4a65-9378-e78592256758-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv\" (UID: \"412ce42d-92e1-4a65-9378-e78592256758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"
Oct 29 23:09:40.965346 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.965216    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brd5x\" (UniqueName: \"kubernetes.io/projected/412ce42d-92e1-4a65-9378-e78592256758-kube-api-access-brd5x\") pod \"maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv\" (UID: \"412ce42d-92e1-4a65-9378-e78592256758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"
Oct 29 23:09:40.965346 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.965253    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/412ce42d-92e1-4a65-9378-e78592256758-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv\" (UID: \"412ce42d-92e1-4a65-9378-e78592256758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"
Oct 29 23:09:40.965346 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.965319    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412ce42d-92e1-4a65-9378-e78592256758-metrics-certs\") pod \"maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv\" (UID: \"412ce42d-92e1-4a65-9378-e78592256758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"
Oct 29 23:09:40.965919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.965891    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/412ce42d-92e1-4a65-9378-e78592256758-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv\" (UID: \"412ce42d-92e1-4a65-9378-e78592256758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"
Oct 29 23:09:40.967652 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.967627    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/412ce42d-92e1-4a65-9378-e78592256758-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv\" (UID: \"412ce42d-92e1-4a65-9378-e78592256758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"
Oct 29 23:09:40.967768 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.967673    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412ce42d-92e1-4a65-9378-e78592256758-metrics-certs\") pod \"maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv\" (UID: \"412ce42d-92e1-4a65-9378-e78592256758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"
Oct 29 23:09:40.989140 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.989115    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brd5x\" (UniqueName: \"kubernetes.io/projected/412ce42d-92e1-4a65-9378-e78592256758-kube-api-access-brd5x\") pod \"maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv\" (UID: \"412ce42d-92e1-4a65-9378-e78592256758\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"
Oct 29 23:09:40.996674 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.996651    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"
Oct 29 23:09:40.999729 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:40.999704    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:41.316019 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:41.315991    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"]
Oct 29 23:09:41.317362 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:09:41.317329    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412ce42d_92e1_4a65_9378_e78592256758.slice/crio-814e861ca175d6c059c499ee987093b29229548b0e0a038d6241d6cd28179d74 WatchSource:0}: Error finding container 814e861ca175d6c059c499ee987093b29229548b0e0a038d6241d6cd28179d74: Status 404 returned error can't find the container with id 814e861ca175d6c059c499ee987093b29229548b0e0a038d6241d6cd28179d74
Oct 29 23:09:41.852258 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:41.852224    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv" event={"ID":"412ce42d-92e1-4a65-9378-e78592256758","Type":"ContainerStarted","Data":"056e8d2f9c1a5a313a643160f7fe86741917fba52e552221ccbfeebc1545bcf5"}
Oct 29 23:09:41.852690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:41.852278    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv" event={"ID":"412ce42d-92e1-4a65-9378-e78592256758","Type":"ContainerStarted","Data":"814e861ca175d6c059c499ee987093b29229548b0e0a038d6241d6cd28179d74"}
Oct 29 23:09:41.852690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:41.852438    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"
Oct 29 23:09:41.898428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:41.898320    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv" podStartSLOduration=1.576221709 podStartE2EDuration="1.898297729s" totalImagesPullingTime="322.07602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:41.89519118 +0000 UTC m=+1585.238139926" watchObservedRunningTime="2026-10-29 23:09:41.898297729 +0000 UTC m=+1585.241246478"
Oct 29 23:09:42.777011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:42.776981    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:42.780105 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:42.780080    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85bd1ee8-788b-492f-a1ed-e09be149f040" path="/var/lib/kubelet/pods/85bd1ee8-788b-492f-a1ed-e09be149f040/volumes"
Oct 29 23:09:44.909147 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:44.909111    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:46.583858 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:46.583827    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:47.367735 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:09:47.367692    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:09:47.866223 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:47.866196    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"
Oct 29 23:09:48.870498 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:48.870462    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:50.580905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:50.580875    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:52.341586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:52.341552    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:54.147486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:54.147452    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:55.967633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:55.967567    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:57.662383 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:57.662350    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:58.886730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:58.886690    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:59.040085 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:09:59.040047    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:00.169785 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:00.169752    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:01.966630 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:01.966573    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:03.861625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:03.861591    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:07.859920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:07.859884    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:09.149381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:09.149345    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:11.973500 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:11.973457    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:12.946863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:12.946831    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-822k7_5d767414-e11d-4113-90ee-f2f0ae1588a1/payload-processing/0.log"
Oct 29 23:10:13.164273 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:13.164245    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-822k7_5d767414-e11d-4113-90ee-f2f0ae1588a1/payload-processing/0.log"
Oct 29 23:10:15.155359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:15.155325    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:15.873639 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:15.873605    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"]
Oct 29 23:10:15.873955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:15.873821    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7" podUID="d4118f7d-454f-4162-8bb5-f58214106d7d" containerName="discovery" containerID="cri-o://29004051e9ce3d93ba8a32a9824171f0195861091decd27be2504eb4faf4e928" gracePeriod=30
Oct 29 23:10:16.158402 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.158375    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:10:16.175914 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.175882    5428 generic.go:356] "Generic (PLEG): container finished" podID="d4118f7d-454f-4162-8bb5-f58214106d7d" containerID="29004051e9ce3d93ba8a32a9824171f0195861091decd27be2504eb4faf4e928" exitCode=0
Oct 29 23:10:16.176086 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.175929    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"
Oct 29 23:10:16.176086 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.175967    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7" event={"ID":"d4118f7d-454f-4162-8bb5-f58214106d7d","Type":"ContainerDied","Data":"29004051e9ce3d93ba8a32a9824171f0195861091decd27be2504eb4faf4e928"}
Oct 29 23:10:16.176086 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.176021    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7" event={"ID":"d4118f7d-454f-4162-8bb5-f58214106d7d","Type":"ContainerDied","Data":"42335d7d0649eaa881569daf581c985a8556258fd52e1ca17ad81d76d1e2c7fb"}
Oct 29 23:10:16.176086 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.176045    5428 scope.go:122] "RemoveContainer" containerID="29004051e9ce3d93ba8a32a9824171f0195861091decd27be2504eb4faf4e928"
Oct 29 23:10:16.187829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.187792    5428 scope.go:122] "RemoveContainer" containerID="29004051e9ce3d93ba8a32a9824171f0195861091decd27be2504eb4faf4e928"
Oct 29 23:10:16.188119 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:10:16.188098    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29004051e9ce3d93ba8a32a9824171f0195861091decd27be2504eb4faf4e928\": container with ID starting with 29004051e9ce3d93ba8a32a9824171f0195861091decd27be2504eb4faf4e928 not found: ID does not exist" containerID="29004051e9ce3d93ba8a32a9824171f0195861091decd27be2504eb4faf4e928"
Oct 29 23:10:16.188183 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.188127    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29004051e9ce3d93ba8a32a9824171f0195861091decd27be2504eb4faf4e928"} err="failed to get container status \"29004051e9ce3d93ba8a32a9824171f0195861091decd27be2504eb4faf4e928\": rpc error: code = NotFound desc = could not find container \"29004051e9ce3d93ba8a32a9824171f0195861091decd27be2504eb4faf4e928\": container with ID starting with 29004051e9ce3d93ba8a32a9824171f0195861091decd27be2504eb4faf4e928 not found: ID does not exist"
Oct 29 23:10:16.300228 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.300193    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-csr-dns-cert\") pod \"d4118f7d-454f-4162-8bb5-f58214106d7d\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") "
Oct 29 23:10:16.300411 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.300250    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-cacerts\" (UniqueName: \"kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-cacerts\") pod \"d4118f7d-454f-4162-8bb5-f58214106d7d\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") "
Oct 29 23:10:16.300411 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.300370    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4118f7d-454f-4162-8bb5-f58214106d7d-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4118f7d-454f-4162-8bb5-f58214106d7d-local-certs\") pod \"d4118f7d-454f-4162-8bb5-f58214106d7d\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") "
Oct 29 23:10:16.300542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.300443    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-token\" (UniqueName: \"kubernetes.io/projected/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-token\") pod \"d4118f7d-454f-4162-8bb5-f58214106d7d\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") "
Oct 29 23:10:16.300668 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.300650    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-csr-ca-configmap\") pod \"d4118f7d-454f-4162-8bb5-f58214106d7d\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") "
Oct 29 23:10:16.300757 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.300724    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4118f7d-454f-4162-8bb5-f58214106d7d-kube-api-access-2w5tk\" (UniqueName: \"kubernetes.io/projected/d4118f7d-454f-4162-8bb5-f58214106d7d-kube-api-access-2w5tk\") pod \"d4118f7d-454f-4162-8bb5-f58214106d7d\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") "
Oct 29 23:10:16.300828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.300771    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-kubeconfig\") pod \"d4118f7d-454f-4162-8bb5-f58214106d7d\" (UID: \"d4118f7d-454f-4162-8bb5-f58214106d7d\") "
Oct 29 23:10:16.301426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.301121    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-csr-ca-configmap" pod "d4118f7d-454f-4162-8bb5-f58214106d7d" (UID: "d4118f7d-454f-4162-8bb5-f58214106d7d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:10:16.301561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.301445    5428 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-csr-ca-configmap\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:10:16.303048 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.303024    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-csr-dns-cert" pod "d4118f7d-454f-4162-8bb5-f58214106d7d" (UID: "d4118f7d-454f-4162-8bb5-f58214106d7d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:10:16.303148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.303022    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4118f7d-454f-4162-8bb5-f58214106d7d-local-certs" pod "d4118f7d-454f-4162-8bb5-f58214106d7d" (UID: "d4118f7d-454f-4162-8bb5-f58214106d7d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:10:16.303148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.303056    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-token" pod "d4118f7d-454f-4162-8bb5-f58214106d7d" (UID: "d4118f7d-454f-4162-8bb5-f58214106d7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:10:16.303432 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.303405    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-cacerts" pod "d4118f7d-454f-4162-8bb5-f58214106d7d" (UID: "d4118f7d-454f-4162-8bb5-f58214106d7d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:10:16.303553 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.303531    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-kubeconfig" pod "d4118f7d-454f-4162-8bb5-f58214106d7d" (UID: "d4118f7d-454f-4162-8bb5-f58214106d7d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:10:16.303613 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.303578    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4118f7d-454f-4162-8bb5-f58214106d7d-kube-api-access-2w5tk" pod "d4118f7d-454f-4162-8bb5-f58214106d7d" (UID: "d4118f7d-454f-4162-8bb5-f58214106d7d"). InnerVolumeSpecName "kube-api-access-2w5tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:10:16.403107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.403030    5428 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-csr-dns-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:10:16.403107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.403059    5428 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-cacerts\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:10:16.403107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.403070    5428 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4118f7d-454f-4162-8bb5-f58214106d7d-local-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:10:16.403107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.403079    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:10:16.403107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.403089    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w5tk\" (UniqueName: \"kubernetes.io/projected/d4118f7d-454f-4162-8bb5-f58214106d7d-kube-api-access-2w5tk\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:10:16.403107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.403097    5428 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4118f7d-454f-4162-8bb5-f58214106d7d-istio-kubeconfig\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:10:16.525788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.525753    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"]
Oct 29 23:10:16.537587 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.537557    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmvx7"]
Oct 29 23:10:16.779760 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:16.779677    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4118f7d-454f-4162-8bb5-f58214106d7d" path="/var/lib/kubelet/pods/d4118f7d-454f-4162-8bb5-f58214106d7d/volumes"
Oct 29 23:10:17.278684 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:17.278651    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:17.370693 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:10:17.370656    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:10:17.818700 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:17.818667    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:19.714749 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:19.714712    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:20.581312 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:20.581280    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:20.699327 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:20.699297    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:21.866379 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:21.866350    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:22.262328 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:22.262300    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:23.468502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:23.468470    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:23.918008 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:23.917967    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"]
Oct 29 23:10:23.919072 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:23.919045    5428 cpu_manager.go:414] "containerMap: removing container" podUID="d4118f7d-454f-4162-8bb5-f58214106d7d" containerName="discovery"
Oct 29 23:10:23.919155 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:23.919077    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4118f7d-454f-4162-8bb5-f58214106d7d" containerName="discovery"
Oct 29 23:10:23.919266 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:23.919252    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4118f7d-454f-4162-8bb5-f58214106d7d" containerName="discovery"
Oct 29 23:10:23.925174 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:23.925147    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:23.941099 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:23.941078    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-65ffcb-openshift-default-dockercfg-84vwv\""
Oct 29 23:10:23.957997 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:23.957963    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"]
Oct 29 23:10:23.993991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:23.993958    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:24.004614 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.004587    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-data\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.004762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.004624    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-podinfo\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.004762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.004662    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-envoy\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.004762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.004750    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-token\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.004906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.004867    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-workload-socket\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.004947 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.004915    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-credential-socket\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.004947 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.004935    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-workload-certs\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.005086 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.005073    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854e3883-53e9-4c18-ac91-bba8f9d4987e-istiod-ca-cert\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.005126 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.005096    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjhd7\" (UniqueName: \"kubernetes.io/projected/854e3883-53e9-4c18-ac91-bba8f9d4987e-kube-api-access-rjhd7\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.106604 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.106557    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjhd7\" (UniqueName: \"kubernetes.io/projected/854e3883-53e9-4c18-ac91-bba8f9d4987e-kube-api-access-rjhd7\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.106848 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.106617    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-data\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.106848 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.106663    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-podinfo\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.106848 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.106723    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-envoy\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.106848 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.106796    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-token\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.107103 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.106998    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-workload-socket\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.107103 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.107066    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-credential-socket\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.107103 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.107094    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-workload-certs\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.107262 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.107112    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-data\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.107262 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.107247    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854e3883-53e9-4c18-ac91-bba8f9d4987e-istiod-ca-cert\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.107439 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.107419    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-workload-certs\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.107755 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.107727    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-workload-socket\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.107979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.107953    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-credential-socket\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.108047 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.108004    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854e3883-53e9-4c18-ac91-bba8f9d4987e-istiod-ca-cert\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.110126 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.110098    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-podinfo\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.110225 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.110183    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-envoy\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.130211 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.130173    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-token\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.130620 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.130594    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjhd7\" (UniqueName: \"kubernetes.io/projected/854e3883-53e9-4c18-ac91-bba8f9d4987e-kube-api-access-rjhd7\") pod \"e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") " pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.139467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.139434    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:24.241613 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.241530    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:24.252693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.252663    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"]
Oct 29 23:10:24.259345 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.259321    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.284668 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.284635    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"]
Oct 29 23:10:24.291409 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.291371    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"]
Oct 29 23:10:24.297634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.297497    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.330294 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.329890    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"]
Oct 29 23:10:24.412624 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.412583    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwcnb\" (UniqueName: \"kubernetes.io/projected/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-kube-api-access-kwcnb\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.412820 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.412687    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.412820 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.412730    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.412820 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.412757    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.412985 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.412863    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.412985 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.412901    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.412985 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.412937    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.413146 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.413127    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.413459 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.413438    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.413513 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.413490    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.413593 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.413515    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.413593 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.413532    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db52cef6-d02d-439a-97ee-685acbad592d-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.413593 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.413565    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvs2x\" (UniqueName: \"kubernetes.io/projected/db52cef6-d02d-439a-97ee-685acbad592d-kube-api-access-wvs2x\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.413689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.413615    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.413689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.413634    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.413755 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.413705    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db52cef6-d02d-439a-97ee-685acbad592d-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.413755 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.413738    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.413861 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.413764    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db52cef6-d02d-439a-97ee-685acbad592d-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.514860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.514726    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.514860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.514769    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.514860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.514856    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db52cef6-d02d-439a-97ee-685acbad592d-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.515453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.514897    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.515453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.514937    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db52cef6-d02d-439a-97ee-685acbad592d-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.515453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.514967    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwcnb\" (UniqueName: \"kubernetes.io/projected/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-kube-api-access-kwcnb\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.515453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.514996    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.515453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.515020    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.515453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.515043    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.515453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.515098    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.515453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.515134    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.515453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.515169    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.515453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.515192    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.515453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.515287    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.515453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.515342    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.515453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.515350    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.515453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.515455    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.516215 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.515491    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.516215 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.515496    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.516215 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.515526    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db52cef6-d02d-439a-97ee-685acbad592d-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.516215 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.515580    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvs2x\" (UniqueName: \"kubernetes.io/projected/db52cef6-d02d-439a-97ee-685acbad592d-kube-api-access-wvs2x\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.516215 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.515651    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-workload-certs\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.516215 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.515717    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.516215 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.515747    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db52cef6-d02d-439a-97ee-685acbad592d-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.516215 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.515794    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-credential-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.516215 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.516086    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-istio-data\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.516706 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.516306    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-workload-socket\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.516706 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.516353    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istiod-ca-cert\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.517916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.517887    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.518136 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.518106    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.518243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.518218    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-envoy\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.518421 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.518405    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db52cef6-d02d-439a-97ee-685acbad592d-istio-podinfo\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.540711 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.540677    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db52cef6-d02d-439a-97ee-685acbad592d-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.540892 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.540739    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvs2x\" (UniqueName: \"kubernetes.io/projected/db52cef6-d02d-439a-97ee-685acbad592d-kube-api-access-wvs2x\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.540892 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.540748    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-token\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.540892 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.540889    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwcnb\" (UniqueName: \"kubernetes.io/projected/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-kube-api-access-kwcnb\") pod \"e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") " pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.603341 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.603316    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"]
Oct 29 23:10:24.604028 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:10:24.604001    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854e3883_53e9_4c18_ac91_bba8f9d4987e.slice/crio-eb7fd5512bd01a8e1ee8cd0962e31296dc047a893280eb9759b20c9ae37e5b2a WatchSource:0}: Error finding container eb7fd5512bd01a8e1ee8cd0962e31296dc047a893280eb9759b20c9ae37e5b2a: Status 404 returned error can't find the container with id eb7fd5512bd01a8e1ee8cd0962e31296dc047a893280eb9759b20c9ae37e5b2a
Oct 29 23:10:24.604872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.604852    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:24.606668 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.606635    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:10:24.606748 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.606700    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:10:24.606788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.606746    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:10:24.636281 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.636258    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:24.801056 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.790137    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:24.942293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.942265    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"]
Oct 29 23:10:24.943061 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:10:24.943037    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb52cef6_d02d_439a_97ee_685acbad592d.slice/crio-0a5aa95b1e702fab028998b493bb0479fb5e1321c93c792b224f837db7f161c0 WatchSource:0}: Error finding container 0a5aa95b1e702fab028998b493bb0479fb5e1321c93c792b224f837db7f161c0: Status 404 returned error can't find the container with id 0a5aa95b1e702fab028998b493bb0479fb5e1321c93c792b224f837db7f161c0
Oct 29 23:10:24.945965 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.945933    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:10:24.946098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.945998    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:10:24.946098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.946044    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:10:24.962824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.962785    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"]
Oct 29 23:10:24.965022 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:10:24.964994    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b154cd7_0f58_4506_b7ac_2eed207e4a0b.slice/crio-c1dcd1a76817397bac6191a73c2be131254c9ca5ae188d6b495d971934aed15a WatchSource:0}: Error finding container c1dcd1a76817397bac6191a73c2be131254c9ca5ae188d6b495d971934aed15a: Status 404 returned error can't find the container with id c1dcd1a76817397bac6191a73c2be131254c9ca5ae188d6b495d971934aed15a
Oct 29 23:10:24.968481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.968416    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:10:24.968610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.968504    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:10:24.968610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:24.968551    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:10:25.251520 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:25.251482    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn" event={"ID":"db52cef6-d02d-439a-97ee-685acbad592d","Type":"ContainerStarted","Data":"a35be425041751b7e9fce7961305479a13fa689037c1b21fcecde5c3f9d1869b"}
Oct 29 23:10:25.251758 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:25.251743    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn" event={"ID":"db52cef6-d02d-439a-97ee-685acbad592d","Type":"ContainerStarted","Data":"0a5aa95b1e702fab028998b493bb0479fb5e1321c93c792b224f837db7f161c0"}
Oct 29 23:10:25.254960 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:25.254927    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf" event={"ID":"1b154cd7-0f58-4506-b7ac-2eed207e4a0b","Type":"ContainerStarted","Data":"c7e5ffafe624e7674aed5430da116b976d2d15375838c0e4d94ee33cc9421d75"}
Oct 29 23:10:25.255320 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:25.255285    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf" event={"ID":"1b154cd7-0f58-4506-b7ac-2eed207e4a0b","Type":"ContainerStarted","Data":"c1dcd1a76817397bac6191a73c2be131254c9ca5ae188d6b495d971934aed15a"}
Oct 29 23:10:25.257069 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:25.257044    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w" event={"ID":"854e3883-53e9-4c18-ac91-bba8f9d4987e","Type":"ContainerStarted","Data":"fa3a7376c623bfade6df0807bb3d645174ccd8ed0a5e783be3e71798832e4608"}
Oct 29 23:10:25.257236 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:25.257218    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w" event={"ID":"854e3883-53e9-4c18-ac91-bba8f9d4987e","Type":"ContainerStarted","Data":"eb7fd5512bd01a8e1ee8cd0962e31296dc047a893280eb9759b20c9ae37e5b2a"}
Oct 29 23:10:25.299481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:25.299420    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn" podStartSLOduration=1.299403457 podStartE2EDuration="1.299403457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:25.296633118 +0000 UTC m=+1628.639581889" watchObservedRunningTime="2026-10-29 23:10:25.299403457 +0000 UTC m=+1628.642352204"
Oct 29 23:10:25.347466 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:25.347356    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w" podStartSLOduration=2.347336647 podStartE2EDuration="2.347336647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:25.3473351 +0000 UTC m=+1628.690283867" watchObservedRunningTime="2026-10-29 23:10:25.347336647 +0000 UTC m=+1628.690285393"
Oct 29 23:10:25.396394 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:25.396325    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf" podStartSLOduration=1.396306896 podStartE2EDuration="1.396306896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:25.392275796 +0000 UTC m=+1628.735224542" watchObservedRunningTime="2026-10-29 23:10:25.396306896 +0000 UTC m=+1628.739255642"
Oct 29 23:10:25.605315 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:25.605176    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:25.636662 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:25.636628    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:26.008826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.005701    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"]
Oct 29 23:10:26.049828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.042000    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"]
Oct 29 23:10:26.049828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.042051    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"]
Oct 29 23:10:26.050428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.050405    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.064833 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.058003    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"]
Oct 29 23:10:26.064833 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.058121    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.151176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.151144    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.151302 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.151182    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.151302 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.151206    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.151382 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.151300    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.151382 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.151353    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.151382 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.151372    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.151485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.151388    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.151485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.151457    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.151485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.151483    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.151619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.151509    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95wft\" (UniqueName: \"kubernetes.io/projected/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-kube-api-access-95wft\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.151619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.151528    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.151619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.151548    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eef2e22-9704-4291-b7ae-91c29acd11de-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.151726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.151637    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.151726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.151657    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.151726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.151675    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.151847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.151745    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.151847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.151775    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.151847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.151830    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v6wm\" (UniqueName: \"kubernetes.io/projected/2eef2e22-9704-4291-b7ae-91c29acd11de-kube-api-access-9v6wm\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.241899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.241868    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:26.248006 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.247981    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:26.252746 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.252722    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.252831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.252761    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.252831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.252790    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.252949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.252921    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.252986 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.252958    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.253056 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.252982    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.253056 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253008    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.253175 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253069    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.253175 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253104    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.253175 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253150    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95wft\" (UniqueName: \"kubernetes.io/projected/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-kube-api-access-95wft\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.253318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253181    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.253318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253207    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eef2e22-9704-4291-b7ae-91c29acd11de-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.253318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253246    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.253318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253257    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.253318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253303    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.253584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253341    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.253584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253360    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.253584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253438    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.253584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253446    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.253584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253505    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-credential-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.253584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253548    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.253584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253566    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-workload-certs\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.253979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253620    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v6wm\" (UniqueName: \"kubernetes.io/projected/2eef2e22-9704-4291-b7ae-91c29acd11de-kube-api-access-9v6wm\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.253979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253691    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.253979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253825    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-workload-socket\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.253979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.253941    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eef2e22-9704-4291-b7ae-91c29acd11de-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.254232 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.254189    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-data\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.254302 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.254196    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istiod-ca-cert\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.256388 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.256362    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.256493 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.256448    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.256608 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.256586    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-envoy\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.256792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.256774    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-podinfo\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.316875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.315875    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95wft\" (UniqueName: \"kubernetes.io/projected/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-kube-api-access-95wft\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.325706 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.322377    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.325706 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.322395    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-token\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.325706 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.323090    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v6wm\" (UniqueName: \"kubernetes.io/projected/2eef2e22-9704-4291-b7ae-91c29acd11de-kube-api-access-9v6wm\") pod \"e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") " pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.325706 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.324700    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:26.326255 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.326194    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:10:26.442297 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.442261    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:26.452251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.452205    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:26.618321 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.612648    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 29 23:10:26.618321 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.612720    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn" podUID="db52cef6-d02d-439a-97ee-685acbad592d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 29 23:10:26.638516 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.638484    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 29 23:10:26.638639 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.638542    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf" podUID="1b154cd7-0f58-4506-b7ac-2eed207e4a0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 29 23:10:26.918278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.918253    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"]
Oct 29 23:10:26.920649 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:10:26.920622    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8f9e56_8bb0_4657_9539_f7b3aa80af15.slice/crio-77f5beec63d4615b272b2830718fcb4207f41577bf6520c9fbb30a3cdda1a5b8 WatchSource:0}: Error finding container 77f5beec63d4615b272b2830718fcb4207f41577bf6520c9fbb30a3cdda1a5b8: Status 404 returned error can't find the container with id 77f5beec63d4615b272b2830718fcb4207f41577bf6520c9fbb30a3cdda1a5b8
Oct 29 23:10:26.923243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.923211    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:10:26.923314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.923277    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:10:26.923314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.923308    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:10:26.936365 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.936342    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"]
Oct 29 23:10:26.940332 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:10:26.940307    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eef2e22_9704_4291_b7ae_91c29acd11de.slice/crio-eccbc26cadcdfd99080872d26968b65106ece9eb177c4ae14b61ad030f87fbe5 WatchSource:0}: Error finding container eccbc26cadcdfd99080872d26968b65106ece9eb177c4ae14b61ad030f87fbe5: Status 404 returned error can't find the container with id eccbc26cadcdfd99080872d26968b65106ece9eb177c4ae14b61ad030f87fbe5
Oct 29 23:10:26.945763 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.945157    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:10:26.945763 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.945218    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:10:26.945763 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:26.945248    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:10:27.141529 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:27.141453    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:27.285881 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:27.267283    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:27.342356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:27.342319    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9" event={"ID":"0a8f9e56-8bb0-4657-9539-f7b3aa80af15","Type":"ContainerStarted","Data":"163e4791592b6453ca1ef9720d3ebf5aef1f657b1af8a1165474d0d01f9ee166"}
Oct 29 23:10:27.342532 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:27.342384    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9" event={"ID":"0a8f9e56-8bb0-4657-9539-f7b3aa80af15","Type":"ContainerStarted","Data":"77f5beec63d4615b272b2830718fcb4207f41577bf6520c9fbb30a3cdda1a5b8"}
Oct 29 23:10:27.346860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:27.346291    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh" event={"ID":"2eef2e22-9704-4291-b7ae-91c29acd11de","Type":"ContainerStarted","Data":"b1f9dff7048aec5eb8ca721e37f148e3fb07db50bab779d76ff357aa43c3bede"}
Oct 29 23:10:27.346860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:27.346337    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh" event={"ID":"2eef2e22-9704-4291-b7ae-91c29acd11de","Type":"ContainerStarted","Data":"eccbc26cadcdfd99080872d26968b65106ece9eb177c4ae14b61ad030f87fbe5"}
Oct 29 23:10:27.423339 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:27.423269    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9" podStartSLOduration=2.423248579 podStartE2EDuration="2.423248579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:27.407367491 +0000 UTC m=+1630.750316267" watchObservedRunningTime="2026-10-29 23:10:27.423248579 +0000 UTC m=+1630.766197325"
Oct 29 23:10:27.442740 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:27.442712    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:27.445649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:27.445619    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:27.452488 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:27.452464    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:27.454956 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:27.454906    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh" podStartSLOduration=2.454893237 podStartE2EDuration="2.454893237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:27.452347329 +0000 UTC m=+1630.795296076" watchObservedRunningTime="2026-10-29 23:10:27.454893237 +0000 UTC m=+1630.797841982"
Oct 29 23:10:27.631264 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:27.631227    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 29 23:10:27.631872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:27.631838    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn" podUID="db52cef6-d02d-439a-97ee-685acbad592d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 29 23:10:27.642266 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:27.642232    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 29 23:10:27.642633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:27.642605    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf" podUID="1b154cd7-0f58-4506-b7ac-2eed207e4a0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 29 23:10:27.919654 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:27.919450    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:28.191000 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:28.190902    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:28.332842 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:28.325195    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:28.455884 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:28.455210    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 29 23:10:28.455884 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:28.455273    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9" podUID="0a8f9e56-8bb0-4657-9539-f7b3aa80af15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 29 23:10:28.471838 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:28.469996    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 29 23:10:28.471838 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:28.470059    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh" podUID="2eef2e22-9704-4291-b7ae-91c29acd11de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 29 23:10:28.615346 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:28.615304    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 29 23:10:28.615612 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:28.615578    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn" podUID="db52cef6-d02d-439a-97ee-685acbad592d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 29 23:10:28.637706 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:28.637664    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 29 23:10:28.638373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:28.638337    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf" podUID="1b154cd7-0f58-4506-b7ac-2eed207e4a0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 29 23:10:29.476824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:29.466215    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 29 23:10:29.476824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:29.466284    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh" podUID="2eef2e22-9704-4291-b7ae-91c29acd11de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 29 23:10:29.499855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:29.499250    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 29 23:10:29.499855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:29.499327    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9" podUID="0a8f9e56-8bb0-4657-9539-f7b3aa80af15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 29 23:10:29.627825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:29.627387    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 29 23:10:29.627825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:29.627458    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn" podUID="db52cef6-d02d-439a-97ee-685acbad592d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 29 23:10:29.640826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:29.638298    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 29 23:10:29.640826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:29.638395    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf" podUID="1b154cd7-0f58-4506-b7ac-2eed207e4a0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 29 23:10:29.766884 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:29.752631    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:30.072567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.072462    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:30.092285 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.092223    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:30.214241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.214204    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:30.297824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.295311    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz"]
Oct 29 23:10:30.298998 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.298962    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:30.332013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.313039    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz"
Oct 29 23:10:30.556675 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.388887    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b3b173c-cc70-45a8-a2c6-7f1321e611c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz\" (UID: \"7b3b173c-cc70-45a8-a2c6-7f1321e611c3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz"
Oct 29 23:10:30.645470 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.645428    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7r27\" (UniqueName: \"kubernetes.io/projected/7b3b173c-cc70-45a8-a2c6-7f1321e611c3-kube-api-access-d7r27\") pod \"payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz\" (UID: \"7b3b173c-cc70-45a8-a2c6-7f1321e611c3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz"
Oct 29 23:10:30.646295 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.437305    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-65ffcb\""
Oct 29 23:10:30.646432 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.436587    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-65ffcb-dockercfg-hgdvv\""
Oct 29 23:10:30.646576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.470070    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 29 23:10:30.646715 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.646689    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh" podUID="2eef2e22-9704-4291-b7ae-91c29acd11de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 29 23:10:30.647196 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.594704    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:30.647364 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.627383    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz"]
Oct 29 23:10:30.647515 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.470387    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 29 23:10:30.647648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.647629    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9" podUID="0a8f9e56-8bb0-4657-9539-f7b3aa80af15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 29 23:10:30.648369 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.630506    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 29 23:10:30.669456 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.669421    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn" podUID="db52cef6-d02d-439a-97ee-685acbad592d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 29 23:10:30.669884 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.669860    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 29 23:10:30.670634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.670600    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf" podUID="1b154cd7-0f58-4506-b7ac-2eed207e4a0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 29 23:10:30.671003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.670982    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:30.728697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.720375    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb"]
Oct 29 23:10:30.738102 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.738061    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:30.744566 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.744542    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb"
Oct 29 23:10:30.762700 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.759785    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7r27\" (UniqueName: \"kubernetes.io/projected/7b3b173c-cc70-45a8-a2c6-7f1321e611c3-kube-api-access-d7r27\") pod \"payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz\" (UID: \"7b3b173c-cc70-45a8-a2c6-7f1321e611c3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz"
Oct 29 23:10:30.762700 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.760045    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b3b173c-cc70-45a8-a2c6-7f1321e611c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz\" (UID: \"7b3b173c-cc70-45a8-a2c6-7f1321e611c3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz"
Oct 29 23:10:30.762700 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.761115    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b3b173c-cc70-45a8-a2c6-7f1321e611c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz\" (UID: \"7b3b173c-cc70-45a8-a2c6-7f1321e611c3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz"
Oct 29 23:10:30.794862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.792361    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:30.892419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.892379    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2def0aa-9f21-412f-ba03-50ccd743d3bb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb\" (UID: \"e2def0aa-9f21-412f-ba03-50ccd743d3bb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb"
Oct 29 23:10:30.896199 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.896123    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fm9t\" (UniqueName: \"kubernetes.io/projected/e2def0aa-9f21-412f-ba03-50ccd743d3bb-kube-api-access-5fm9t\") pod \"payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb\" (UID: \"e2def0aa-9f21-412f-ba03-50ccd743d3bb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb"
Oct 29 23:10:30.903102 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:30.894318    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb"]
Oct 29 23:10:31.037049 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.037016    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7r27\" (UniqueName: \"kubernetes.io/projected/7b3b173c-cc70-45a8-a2c6-7f1321e611c3-kube-api-access-d7r27\") pod \"payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz\" (UID: \"7b3b173c-cc70-45a8-a2c6-7f1321e611c3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz"
Oct 29 23:10:31.038634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.038607    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2def0aa-9f21-412f-ba03-50ccd743d3bb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb\" (UID: \"e2def0aa-9f21-412f-ba03-50ccd743d3bb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb"
Oct 29 23:10:31.038917 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.038897    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fm9t\" (UniqueName: \"kubernetes.io/projected/e2def0aa-9f21-412f-ba03-50ccd743d3bb-kube-api-access-5fm9t\") pod \"payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb\" (UID: \"e2def0aa-9f21-412f-ba03-50ccd743d3bb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb"
Oct 29 23:10:31.062829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.059070    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2def0aa-9f21-412f-ba03-50ccd743d3bb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb\" (UID: \"e2def0aa-9f21-412f-ba03-50ccd743d3bb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb"
Oct 29 23:10:31.108068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.104675    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fm9t\" (UniqueName: \"kubernetes.io/projected/e2def0aa-9f21-412f-ba03-50ccd743d3bb-kube-api-access-5fm9t\") pod \"payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb\" (UID: \"e2def0aa-9f21-412f-ba03-50ccd743d3bb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb"
Oct 29 23:10:31.147878 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.137903    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:31.147878 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.138568    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 23:10:31.219823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.219162    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb"
Oct 29 23:10:31.289841 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.274384    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz"
Oct 29 23:10:31.299607 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.299565    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:31.321870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.300247    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 23:10:31.607986 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.607947    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 29 23:10:31.608309 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.608274    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh" podUID="2eef2e22-9704-4291-b7ae-91c29acd11de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 29 23:10:31.637155 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.608505    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 29 23:10:31.639457 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.639412    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9" podUID="0a8f9e56-8bb0-4657-9539-f7b3aa80af15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 29 23:10:31.732880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.693978    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 29 23:10:31.732880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.732623    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn" podUID="db52cef6-d02d-439a-97ee-685acbad592d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 29 23:10:31.776857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.760915    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 29 23:10:31.776857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.763884    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:31.776857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.764916    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 23:10:31.803914 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.778873    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 29 23:10:31.817053 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.817004    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf" podUID="1b154cd7-0f58-4506-b7ac-2eed207e4a0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 29 23:10:31.822450 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.822407    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:31.852396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.851330    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 23:10:31.865563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.864712    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 23:10:31.956326 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:31.956290    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 23:10:32.446451 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:32.446416    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 29 23:10:32.446650 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:32.446470    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9" podUID="0a8f9e56-8bb0-4657-9539-f7b3aa80af15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 29 23:10:32.452821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:32.452696    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 29 23:10:32.452821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:32.452751    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh" podUID="2eef2e22-9704-4291-b7ae-91c29acd11de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 29 23:10:32.606330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:32.606230    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 29 23:10:32.614939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:32.614895    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn" podUID="db52cef6-d02d-439a-97ee-685acbad592d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 29 23:10:32.630182 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:32.630137    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:32.641243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:32.641206    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 29 23:10:32.641501 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:32.641471    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf" podUID="1b154cd7-0f58-4506-b7ac-2eed207e4a0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 29 23:10:33.374879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.344656    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb" event={"ID":"e2def0aa-9f21-412f-ba03-50ccd743d3bb","Type":"ContainerStarted","Data":"73a298b019a8509a83d390a5d23f1494200fd9ad8cf8b8658249023777518435"}
Oct 29 23:10:33.374879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.344726    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb"]
Oct 29 23:10:33.374879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.352842    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz"]
Oct 29 23:10:33.429826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.421876    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"]
Oct 29 23:10:33.533826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.501987    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 29 23:10:33.571854 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.564124    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 29 23:10:33.571854 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.564188    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9" podUID="0a8f9e56-8bb0-4657-9539-f7b3aa80af15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 29 23:10:33.573737 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.573695    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh" podUID="2eef2e22-9704-4291-b7ae-91c29acd11de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 29 23:10:33.729825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.729128    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"]
Oct 29 23:10:33.729825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.729291    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.800877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.799742    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.800877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.799822    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.800877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.799867    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.800877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.799966    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.800877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.800037    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6rwk\" (UniqueName: \"kubernetes.io/projected/76bcb1e7-3c9e-4222-abc4-96c5c461707c-kube-api-access-n6rwk\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.800877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.800095    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.800877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.800141    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.800877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.800260    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.800877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.800322    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.849407 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.849368    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:33.917062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.917030    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.917350 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.917328    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6rwk\" (UniqueName: \"kubernetes.io/projected/76bcb1e7-3c9e-4222-abc4-96c5c461707c-kube-api-access-n6rwk\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.917520 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.917503    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.917625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.917614    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.917837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.917821    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.918013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.917995    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.918153 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.918136    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.918286 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.918271    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.918412 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.918398    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.922596 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.922010    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-podinfo\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.943103 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.939728    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-credential-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.943103 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.940261    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-data\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.943103 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.940709    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-workload-certs\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.943103 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.941394    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istiod-ca-cert\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.968950 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.968907    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 23:10:33.996824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.974965    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-workload-socket\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:33.996824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:33.984141    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:10:34.013996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.003416    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-token\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:34.013996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.005627    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-envoy\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:34.053823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.045482    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:34.146051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.131534    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6rwk\" (UniqueName: \"kubernetes.io/projected/76bcb1e7-3c9e-4222-abc4-96c5c461707c-kube-api-access-n6rwk\") pod \"e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") " pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:34.316826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.308783    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:34.433825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.432383    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz" event={"ID":"7b3b173c-cc70-45a8-a2c6-7f1321e611c3","Type":"ContainerStarted","Data":"a07ceec7156cfcbcd10778e848e03590829d3a2fca6a173716874c0978cb22d9"}
Oct 29 23:10:34.455836 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.454487    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 29 23:10:34.455836 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.454545    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9" podUID="0a8f9e56-8bb0-4657-9539-f7b3aa80af15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 29 23:10:34.455836 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.455236    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 29 23:10:34.455836 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.455271    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh" podUID="2eef2e22-9704-4291-b7ae-91c29acd11de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 29 23:10:34.489838 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.487704    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb" event={"ID":"e2def0aa-9f21-412f-ba03-50ccd743d3bb","Type":"ContainerStarted","Data":"d82b04aa763d9757e0d063307dc850ec31896034e38209300b2db353bf245222"}
Oct 29 23:10:34.489838 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.488755    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb"
Oct 29 23:10:34.514893 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.513995    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:34.518029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.515364    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:34.529392 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.529276    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:10:34.544564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.544487    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb" podStartSLOduration=4.544467288 podStartE2EDuration="4.544467288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:34.539741293 +0000 UTC m=+1637.882690040" watchObservedRunningTime="2026-10-29 23:10:34.544467288 +0000 UTC m=+1637.887416033"
Oct 29 23:10:34.551345 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.551261    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:34.605145 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.604021    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:34.605145 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.604978    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:34.625856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.621227    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:10:34.682562 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:34.671557    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:35.164889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:35.164673    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"]
Oct 29 23:10:35.207655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:35.206575    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:10:35.207655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:35.206674    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:10:35.207655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:35.206745    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:10:35.444599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:35.444517    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 29 23:10:35.444599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:35.444576    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9" podUID="0a8f9e56-8bb0-4657-9539-f7b3aa80af15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 29 23:10:35.456460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:35.455893    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 29 23:10:35.456460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:35.455948    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh" podUID="2eef2e22-9704-4291-b7ae-91c29acd11de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 29 23:10:35.507851 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:35.503900    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz" event={"ID":"7b3b173c-cc70-45a8-a2c6-7f1321e611c3","Type":"ContainerStarted","Data":"60d98cdac4cc9b90765637dc42ab4c565106d0f1152a75664e8e0d1184dfb734"}
Oct 29 23:10:35.507851 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:35.505099    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz"
Oct 29 23:10:35.511501 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:35.511472    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4" event={"ID":"76bcb1e7-3c9e-4222-abc4-96c5c461707c","Type":"ContainerStarted","Data":"d19f9adf5eac8ecc02df18d38a90ff407b917ad2dc692d7530f63a313b95fc43"}
Oct 29 23:10:35.511599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:35.511521    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4" event={"ID":"76bcb1e7-3c9e-4222-abc4-96c5c461707c","Type":"ContainerStarted","Data":"e731972e429b4d80c3b992ded9fdf15bffb5ff6f6dc34b2ade3edf7066ace588"}
Oct 29 23:10:35.689986 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:35.689919    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz" podStartSLOduration=5.68989718 podStartE2EDuration="5.68989718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:35.590638387 +0000 UTC m=+1638.933587184" watchObservedRunningTime="2026-10-29 23:10:35.68989718 +0000 UTC m=+1639.032845944"
Oct 29 23:10:35.690777 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:35.690728    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4" podStartSLOduration=2.690714663 podStartE2EDuration="2.690714663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:35.689330208 +0000 UTC m=+1639.032278979" watchObservedRunningTime="2026-10-29 23:10:35.690714663 +0000 UTC m=+1639.033663409"
Oct 29 23:10:36.311026 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:36.310908    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:36.312289 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:36.312259    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 29 23:10:36.312415 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:36.312320    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4" podUID="76bcb1e7-3c9e-4222-abc4-96c5c461707c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 29 23:10:36.882041 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:36.880138    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:36.882041 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:36.881742    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:36.891141 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:36.891102    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:10:37.209078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:37.208989    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:37.209375 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:37.209354    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:37.210139 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:37.210119    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:10:37.248866 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:37.248830    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:37.311172 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:37.311114    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 29 23:10:37.311386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:37.311210    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4" podUID="76bcb1e7-3c9e-4222-abc4-96c5c461707c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 29 23:10:38.311554 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:38.311518    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 29 23:10:38.312003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:38.311579    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4" podUID="76bcb1e7-3c9e-4222-abc4-96c5c461707c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 29 23:10:39.311270 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:39.311236    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 29 23:10:39.311453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:39.311287    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4" podUID="76bcb1e7-3c9e-4222-abc4-96c5c461707c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 29 23:10:39.507125 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:39.507075    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:40.311148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:40.311112    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 29 23:10:40.311338 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:40.311164    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4" podUID="76bcb1e7-3c9e-4222-abc4-96c5c461707c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 29 23:10:41.247651 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:41.247616    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:41.311601 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:41.311561    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 29 23:10:41.311754 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:41.311628    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4" podUID="76bcb1e7-3c9e-4222-abc4-96c5c461707c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 29 23:10:41.396821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:41.396751    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:41.623664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:41.623623    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:42.310819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:42.310771    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 29 23:10:42.311247 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:42.310858    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4" podUID="76bcb1e7-3c9e-4222-abc4-96c5c461707c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 29 23:10:43.273076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:43.273014    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:43.320841 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:43.320767    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:43.321771 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:43.321715    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:43.330974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:43.330940    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:10:45.360523 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:45.360490    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:45.516108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:45.516082    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb"
Oct 29 23:10:47.371231 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:10:47.371191    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:10:47.530684 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:47.530653    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz"
Oct 29 23:10:48.073648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:48.073615    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:49.807191 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:49.807157    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"]
Oct 29 23:10:49.807639 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:49.807349    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv" podUID="0ff6b960-7094-4c65-b7d8-17bae2645ccb" containerName="maas-api" containerID="cri-o://f2293586bc5bbd9d75856714978e8b9d3ace9ed83beaa11df6476c3d2c00ed6a" gracePeriod=30
Oct 29 23:10:49.877473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:49.877439    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:50.067025 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.066925    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:50.168959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.168931    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"
Oct 29 23:10:50.186510 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.186484    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:50.257296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.257258    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ff6b960-7094-4c65-b7d8-17bae2645ccb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff6b960-7094-4c65-b7d8-17bae2645ccb-metrics-certs\") pod \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\" (UID: \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\") "
Oct 29 23:10:50.257482 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.257403    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff6b960-7094-4c65-b7d8-17bae2645ccb-kube-api-access-25q7z\" (UniqueName: \"kubernetes.io/projected/0ff6b960-7094-4c65-b7d8-17bae2645ccb-kube-api-access-25q7z\") pod \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\" (UID: \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\") "
Oct 29 23:10:50.257562 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.257537    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ff6b960-7094-4c65-b7d8-17bae2645ccb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ff6b960-7094-4c65-b7d8-17bae2645ccb-maas-api-tls\") pod \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\" (UID: \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\") "
Oct 29 23:10:50.257625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.257578    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff6b960-7094-4c65-b7d8-17bae2645ccb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ff6b960-7094-4c65-b7d8-17bae2645ccb-openshift-service-ca\") pod \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\" (UID: \"0ff6b960-7094-4c65-b7d8-17bae2645ccb\") "
Oct 29 23:10:50.258320 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.258285    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff6b960-7094-4c65-b7d8-17bae2645ccb-openshift-service-ca" pod "0ff6b960-7094-4c65-b7d8-17bae2645ccb" (UID: "0ff6b960-7094-4c65-b7d8-17bae2645ccb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:10:50.258764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.258742    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ff6b960-7094-4c65-b7d8-17bae2645ccb-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:10:50.260006 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.259980    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff6b960-7094-4c65-b7d8-17bae2645ccb-maas-api-tls" pod "0ff6b960-7094-4c65-b7d8-17bae2645ccb" (UID: "0ff6b960-7094-4c65-b7d8-17bae2645ccb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:10:50.260521 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.260500    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff6b960-7094-4c65-b7d8-17bae2645ccb-metrics-certs" pod "0ff6b960-7094-4c65-b7d8-17bae2645ccb" (UID: "0ff6b960-7094-4c65-b7d8-17bae2645ccb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:10:50.260622 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.260564    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff6b960-7094-4c65-b7d8-17bae2645ccb-kube-api-access-25q7z" pod "0ff6b960-7094-4c65-b7d8-17bae2645ccb" (UID: "0ff6b960-7094-4c65-b7d8-17bae2645ccb"). InnerVolumeSpecName "kube-api-access-25q7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:10:50.360379 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.360298    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ff6b960-7094-4c65-b7d8-17bae2645ccb-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:10:50.360379 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.360332    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25q7z\" (UniqueName: \"kubernetes.io/projected/0ff6b960-7094-4c65-b7d8-17bae2645ccb-kube-api-access-25q7z\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:10:50.360379 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.360341    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ff6b960-7094-4c65-b7d8-17bae2645ccb-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:10:50.660075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.660034    5428 generic.go:356] "Generic (PLEG): container finished" podID="0ff6b960-7094-4c65-b7d8-17bae2645ccb" containerID="f2293586bc5bbd9d75856714978e8b9d3ace9ed83beaa11df6476c3d2c00ed6a" exitCode=0
Oct 29 23:10:50.660075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.660081    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"
Oct 29 23:10:50.660352 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.660113    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv" event={"ID":"0ff6b960-7094-4c65-b7d8-17bae2645ccb","Type":"ContainerDied","Data":"f2293586bc5bbd9d75856714978e8b9d3ace9ed83beaa11df6476c3d2c00ed6a"}
Oct 29 23:10:50.660352 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.660181    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv" event={"ID":"0ff6b960-7094-4c65-b7d8-17bae2645ccb","Type":"ContainerDied","Data":"d992a4c88efa0fed6ddba0fb126b4c5b07ed185e52819c01d631cb76f429f623"}
Oct 29 23:10:50.660352 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.660212    5428 scope.go:122] "RemoveContainer" containerID="f2293586bc5bbd9d75856714978e8b9d3ace9ed83beaa11df6476c3d2c00ed6a"
Oct 29 23:10:50.675023 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.674999    5428 scope.go:122] "RemoveContainer" containerID="f2293586bc5bbd9d75856714978e8b9d3ace9ed83beaa11df6476c3d2c00ed6a"
Oct 29 23:10:50.675478 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:10:50.675449    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2293586bc5bbd9d75856714978e8b9d3ace9ed83beaa11df6476c3d2c00ed6a\": container with ID starting with f2293586bc5bbd9d75856714978e8b9d3ace9ed83beaa11df6476c3d2c00ed6a not found: ID does not exist" containerID="f2293586bc5bbd9d75856714978e8b9d3ace9ed83beaa11df6476c3d2c00ed6a"
Oct 29 23:10:50.675581 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.675484    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2293586bc5bbd9d75856714978e8b9d3ace9ed83beaa11df6476c3d2c00ed6a"} err="failed to get container status \"f2293586bc5bbd9d75856714978e8b9d3ace9ed83beaa11df6476c3d2c00ed6a\": rpc error: code = NotFound desc = could not find container \"f2293586bc5bbd9d75856714978e8b9d3ace9ed83beaa11df6476c3d2c00ed6a\": container with ID starting with f2293586bc5bbd9d75856714978e8b9d3ace9ed83beaa11df6476c3d2c00ed6a not found: ID does not exist"
Oct 29 23:10:50.715616 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.715584    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"]
Oct 29 23:10:50.728406 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.728376    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-921680-7b7fd7cdc8-xnhxv"]
Oct 29 23:10:50.752845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.752779    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:50.780047 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.780010    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff6b960-7094-4c65-b7d8-17bae2645ccb" path="/var/lib/kubelet/pods/0ff6b960-7094-4c65-b7d8-17bae2645ccb/volumes"
Oct 29 23:10:50.992761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:50.992644    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:51.105749 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:51.105717    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:51.572973 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:51.572944    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:53.133003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:53.132965    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:55.294149 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:55.294113    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:56.946118 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:56.946080    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"]
Oct 29 23:10:56.946833 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:56.946814    5428 cpu_manager.go:414] "containerMap: removing container" podUID="0ff6b960-7094-4c65-b7d8-17bae2645ccb" containerName="maas-api"
Oct 29 23:10:56.946833 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:56.946834    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff6b960-7094-4c65-b7d8-17bae2645ccb" containerName="maas-api"
Oct 29 23:10:56.946972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:56.946963    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff6b960-7094-4c65-b7d8-17bae2645ccb" containerName="maas-api"
Oct 29 23:10:56.952160 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:56.952140    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"
Oct 29 23:10:56.970059 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:56.970038    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-65ffcb-serving-cert\""
Oct 29 23:10:56.989090 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:56.989001    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"]
Oct 29 23:10:57.045997 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.045964    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:57.058311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.058280    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6\" (UID: \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"
Oct 29 23:10:57.058456 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.058347    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-metrics-certs\") pod \"maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6\" (UID: \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"
Oct 29 23:10:57.058599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.058582    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6\" (UID: \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"
Oct 29 23:10:57.058638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.058614    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m55x2\" (UniqueName: \"kubernetes.io/projected/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-kube-api-access-m55x2\") pod \"maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6\" (UID: \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"
Oct 29 23:10:57.160753 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.160710    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6\" (UID: \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"
Oct 29 23:10:57.161010 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.160781    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m55x2\" (UniqueName: \"kubernetes.io/projected/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-kube-api-access-m55x2\") pod \"maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6\" (UID: \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"
Oct 29 23:10:57.161090 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.161006    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6\" (UID: \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"
Oct 29 23:10:57.161090 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.161062    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-metrics-certs\") pod \"maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6\" (UID: \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"
Oct 29 23:10:57.163035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.163002    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6\" (UID: \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"
Oct 29 23:10:57.173827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.172162    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-metrics-certs\") pod \"maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6\" (UID: \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"
Oct 29 23:10:57.193005 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.188562    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6\" (UID: \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"
Oct 29 23:10:57.201657 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.195746    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:57.201657 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.196621    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m55x2\" (UniqueName: \"kubernetes.io/projected/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-kube-api-access-m55x2\") pod \"maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6\" (UID: \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"
Oct 29 23:10:57.268168 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.268131    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"
Oct 29 23:10:57.492644 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.492565    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx"]
Oct 29 23:10:57.523017 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.522986    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l"]
Oct 29 23:10:57.624513 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.624482    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"]
Oct 29 23:10:57.627448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.627428    5428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:10:57.638459 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.638435    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:57.715876 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.715830    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6" event={"ID":"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a","Type":"ContainerStarted","Data":"b699335fab9fffc0556caf53b6f4bb05aa6176d09529b48ce24792e01718e76f"}
Oct 29 23:10:57.754057 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:57.753989    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:58.212818 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:58.212776    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:58.372736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:58.372700    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:58.509615 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:58.509574    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:58.730117 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:58.730008    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6" event={"ID":"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a","Type":"ContainerStarted","Data":"39600d9ecaeb2aa3f658f72407c5e0f2ae126ce2a36a8c207c1468c18241e861"}
Oct 29 23:10:58.730541 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:58.730513    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"
Oct 29 23:10:58.779824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:58.777768    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6" podStartSLOduration=2.433273402 podStartE2EDuration="2.777754469s" totalImagesPullingTime="344.481067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:58.776745094 +0000 UTC m=+1662.119693865" watchObservedRunningTime="2026-10-29 23:10:58.777754469 +0000 UTC m=+1662.120703213"
Oct 29 23:10:59.891193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:10:59.891160    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:01.890991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:01.890954    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:02.493151 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:02.493108    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx" podUID="b38d4301-782b-4d87-a7f3-3b6036648288" containerName="payload-processing" containerID="cri-o://3c73136f273803a9aa7af3ee5b786a43604500e18798b45b765d5314eb5678c6" gracePeriod=25
Oct 29 23:11:02.523356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:02.523324    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l" podUID="181b8c86-e1c1-4208-9c6b-8a249d58c2a8" containerName="payload-pre-processing" containerID="cri-o://97b0f298efef23c17e595339a6bc3e9333f7bd88c73544b7c7e252c24451ab54" gracePeriod=25
Oct 29 23:11:02.770859 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:02.770823    5428 generic.go:356] "Generic (PLEG): container finished" podID="b38d4301-782b-4d87-a7f3-3b6036648288" containerID="3c73136f273803a9aa7af3ee5b786a43604500e18798b45b765d5314eb5678c6" exitCode=0
Oct 29 23:11:02.781014 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:02.780764    5428 generic.go:356] "Generic (PLEG): container finished" podID="181b8c86-e1c1-4208-9c6b-8a249d58c2a8" containerID="97b0f298efef23c17e595339a6bc3e9333f7bd88c73544b7c7e252c24451ab54" exitCode=0
Oct 29 23:11:02.784148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:02.784112    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx" event={"ID":"b38d4301-782b-4d87-a7f3-3b6036648288","Type":"ContainerDied","Data":"3c73136f273803a9aa7af3ee5b786a43604500e18798b45b765d5314eb5678c6"}
Oct 29 23:11:02.784312 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:02.784183    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l" event={"ID":"181b8c86-e1c1-4208-9c6b-8a249d58c2a8","Type":"ContainerDied","Data":"97b0f298efef23c17e595339a6bc3e9333f7bd88c73544b7c7e252c24451ab54"}
Oct 29 23:11:02.810848 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:02.810820    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx"
Oct 29 23:11:02.865847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:02.865821    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l"
Oct 29 23:11:02.953716 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:02.953682    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b38d4301-782b-4d87-a7f3-3b6036648288-kube-api-access-79s7z\" (UniqueName: \"kubernetes.io/projected/b38d4301-782b-4d87-a7f3-3b6036648288-kube-api-access-79s7z\") pod \"b38d4301-782b-4d87-a7f3-3b6036648288\" (UID: \"b38d4301-782b-4d87-a7f3-3b6036648288\") "
Oct 29 23:11:02.954209 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:02.953742    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181b8c86-e1c1-4208-9c6b-8a249d58c2a8-kube-api-access-bswhv\" (UniqueName: \"kubernetes.io/projected/181b8c86-e1c1-4208-9c6b-8a249d58c2a8-kube-api-access-bswhv\") pod \"181b8c86-e1c1-4208-9c6b-8a249d58c2a8\" (UID: \"181b8c86-e1c1-4208-9c6b-8a249d58c2a8\") "
Oct 29 23:11:02.954209 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:02.953915    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181b8c86-e1c1-4208-9c6b-8a249d58c2a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181b8c86-e1c1-4208-9c6b-8a249d58c2a8-plugins-config-volume\") pod \"181b8c86-e1c1-4208-9c6b-8a249d58c2a8\" (UID: \"181b8c86-e1c1-4208-9c6b-8a249d58c2a8\") "
Oct 29 23:11:02.954209 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:02.954025    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b38d4301-782b-4d87-a7f3-3b6036648288-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38d4301-782b-4d87-a7f3-3b6036648288-plugins-config-volume\") pod \"b38d4301-782b-4d87-a7f3-3b6036648288\" (UID: \"b38d4301-782b-4d87-a7f3-3b6036648288\") "
Oct 29 23:11:02.954373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:02.954328    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181b8c86-e1c1-4208-9c6b-8a249d58c2a8-plugins-config-volume" pod "181b8c86-e1c1-4208-9c6b-8a249d58c2a8" (UID: "181b8c86-e1c1-4208-9c6b-8a249d58c2a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:02.954511 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:02.954481    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38d4301-782b-4d87-a7f3-3b6036648288-plugins-config-volume" pod "b38d4301-782b-4d87-a7f3-3b6036648288" (UID: "b38d4301-782b-4d87-a7f3-3b6036648288"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:02.954734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:02.954714    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181b8c86-e1c1-4208-9c6b-8a249d58c2a8-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:11:02.954774 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:02.954742    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38d4301-782b-4d87-a7f3-3b6036648288-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:11:02.956029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:02.956003    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181b8c86-e1c1-4208-9c6b-8a249d58c2a8-kube-api-access-bswhv" pod "181b8c86-e1c1-4208-9c6b-8a249d58c2a8" (UID: "181b8c86-e1c1-4208-9c6b-8a249d58c2a8"). InnerVolumeSpecName "kube-api-access-bswhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:02.956595 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:02.956566    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38d4301-782b-4d87-a7f3-3b6036648288-kube-api-access-79s7z" pod "b38d4301-782b-4d87-a7f3-3b6036648288" (UID: "b38d4301-782b-4d87-a7f3-3b6036648288"). InnerVolumeSpecName "kube-api-access-79s7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:03.056227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:03.056142    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79s7z\" (UniqueName: \"kubernetes.io/projected/b38d4301-782b-4d87-a7f3-3b6036648288-kube-api-access-79s7z\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:11:03.056227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:03.056171    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bswhv\" (UniqueName: \"kubernetes.io/projected/181b8c86-e1c1-4208-9c6b-8a249d58c2a8-kube-api-access-bswhv\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:11:03.548032 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:03.548001    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:03.790681 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:03.790593    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx"
Oct 29 23:11:03.790681 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:03.790640    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx" event={"ID":"b38d4301-782b-4d87-a7f3-3b6036648288","Type":"ContainerDied","Data":"1d306c87b37e413ec8512f6a4afc7a6efd651940c610b77f2c3f090619888633"}
Oct 29 23:11:03.790941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:03.790710    5428 scope.go:122] "RemoveContainer" containerID="3c73136f273803a9aa7af3ee5b786a43604500e18798b45b765d5314eb5678c6"
Oct 29 23:11:03.792442 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:03.792416    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l"
Oct 29 23:11:03.792567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:03.792445    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l" event={"ID":"181b8c86-e1c1-4208-9c6b-8a249d58c2a8","Type":"ContainerDied","Data":"d5e792de4aad3822008d505dab31d36ab18cda04f330bdba38e2943c84b2dd2f"}
Oct 29 23:11:03.802484 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:03.802445    5428 scope.go:122] "RemoveContainer" containerID="97b0f298efef23c17e595339a6bc3e9333f7bd88c73544b7c7e252c24451ab54"
Oct 29 23:11:03.844168 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:03.844142    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx"]
Oct 29 23:11:03.857240 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:03.857212    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-85c702-6c9c4d875b-f7xmx"]
Oct 29 23:11:03.883495 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:03.883457    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l"]
Oct 29 23:11:03.888782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:03.888752    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-85c702-d65cd4b8c-fb27l"]
Oct 29 23:11:04.744283 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:04.744246    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"
Oct 29 23:11:04.778891 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:04.778865    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="181b8c86-e1c1-4208-9c6b-8a249d58c2a8" path="/var/lib/kubelet/pods/181b8c86-e1c1-4208-9c6b-8a249d58c2a8/volumes"
Oct 29 23:11:04.779359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:04.779336    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b38d4301-782b-4d87-a7f3-3b6036648288" path="/var/lib/kubelet/pods/b38d4301-782b-4d87-a7f3-3b6036648288/volumes"
Oct 29 23:11:05.301425 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:05.301391    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:07.492088 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:07.491971    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:09.192085 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:09.192053    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:10.632695 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:10.632655    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:11.398139 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:11.398091    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:11.544442 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:11.544407    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:11.659686 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:11.659596    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:12.067761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:12.067667    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:13.081928 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.081889    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"]
Oct 29 23:11:13.082437 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.082142    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n" podUID="6112e33d-bd66-4ebc-8fb0-08a595027411" containerName="maas-api" containerID="cri-o://ab6ad1c6ae3faeb9fe06cfac05f3427b2fc6be70f516aa6d2cb93921f260043f" gracePeriod=30
Oct 29 23:11:13.157044 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.157011    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:13.358598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.353430    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:13.389010 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.388985    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"
Oct 29 23:11:13.408334 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.408305    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6112e33d-bd66-4ebc-8fb0-08a595027411-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6112e33d-bd66-4ebc-8fb0-08a595027411-maas-api-tls\") pod \"6112e33d-bd66-4ebc-8fb0-08a595027411\" (UID: \"6112e33d-bd66-4ebc-8fb0-08a595027411\") "
Oct 29 23:11:13.408518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.408405    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6112e33d-bd66-4ebc-8fb0-08a595027411-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6112e33d-bd66-4ebc-8fb0-08a595027411-openshift-service-ca\") pod \"6112e33d-bd66-4ebc-8fb0-08a595027411\" (UID: \"6112e33d-bd66-4ebc-8fb0-08a595027411\") "
Oct 29 23:11:13.408518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.408452    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6112e33d-bd66-4ebc-8fb0-08a595027411-kube-api-access-zxcrj\" (UniqueName: \"kubernetes.io/projected/6112e33d-bd66-4ebc-8fb0-08a595027411-kube-api-access-zxcrj\") pod \"6112e33d-bd66-4ebc-8fb0-08a595027411\" (UID: \"6112e33d-bd66-4ebc-8fb0-08a595027411\") "
Oct 29 23:11:13.408518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.408504    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6112e33d-bd66-4ebc-8fb0-08a595027411-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6112e33d-bd66-4ebc-8fb0-08a595027411-metrics-certs\") pod \"6112e33d-bd66-4ebc-8fb0-08a595027411\" (UID: \"6112e33d-bd66-4ebc-8fb0-08a595027411\") "
Oct 29 23:11:13.408777 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.408750    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6112e33d-bd66-4ebc-8fb0-08a595027411-openshift-service-ca" pod "6112e33d-bd66-4ebc-8fb0-08a595027411" (UID: "6112e33d-bd66-4ebc-8fb0-08a595027411"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:13.409182 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.409163    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6112e33d-bd66-4ebc-8fb0-08a595027411-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:11:13.410740 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.410715    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6112e33d-bd66-4ebc-8fb0-08a595027411-kube-api-access-zxcrj" pod "6112e33d-bd66-4ebc-8fb0-08a595027411" (UID: "6112e33d-bd66-4ebc-8fb0-08a595027411"). InnerVolumeSpecName "kube-api-access-zxcrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:13.410885 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.410869    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6112e33d-bd66-4ebc-8fb0-08a595027411-maas-api-tls" pod "6112e33d-bd66-4ebc-8fb0-08a595027411" (UID: "6112e33d-bd66-4ebc-8fb0-08a595027411"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:11:13.410955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.410934    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6112e33d-bd66-4ebc-8fb0-08a595027411-metrics-certs" pod "6112e33d-bd66-4ebc-8fb0-08a595027411" (UID: "6112e33d-bd66-4ebc-8fb0-08a595027411"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:11:13.470554 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.470517    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:13.510584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.510559    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6112e33d-bd66-4ebc-8fb0-08a595027411-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:11:13.510584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.510584    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxcrj\" (UniqueName: \"kubernetes.io/projected/6112e33d-bd66-4ebc-8fb0-08a595027411-kube-api-access-zxcrj\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:11:13.510761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.510594    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6112e33d-bd66-4ebc-8fb0-08a595027411-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:11:13.821894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.821691    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:13.872573 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.872541    5428 generic.go:356] "Generic (PLEG): container finished" podID="6112e33d-bd66-4ebc-8fb0-08a595027411" containerID="ab6ad1c6ae3faeb9fe06cfac05f3427b2fc6be70f516aa6d2cb93921f260043f" exitCode=0
Oct 29 23:11:13.872765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.872583    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"
Oct 29 23:11:13.872765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.872620    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n" event={"ID":"6112e33d-bd66-4ebc-8fb0-08a595027411","Type":"ContainerDied","Data":"ab6ad1c6ae3faeb9fe06cfac05f3427b2fc6be70f516aa6d2cb93921f260043f"}
Oct 29 23:11:13.872765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.872685    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n" event={"ID":"6112e33d-bd66-4ebc-8fb0-08a595027411","Type":"ContainerDied","Data":"a08cbd7a86592cc6185f346aeeb1eb628cf1fb8209e4ff6be78ecb18189469e4"}
Oct 29 23:11:13.872765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.872718    5428 scope.go:122] "RemoveContainer" containerID="ab6ad1c6ae3faeb9fe06cfac05f3427b2fc6be70f516aa6d2cb93921f260043f"
Oct 29 23:11:13.883660 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.883621    5428 scope.go:122] "RemoveContainer" containerID="ab6ad1c6ae3faeb9fe06cfac05f3427b2fc6be70f516aa6d2cb93921f260043f"
Oct 29 23:11:13.883916 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:11:13.883897    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab6ad1c6ae3faeb9fe06cfac05f3427b2fc6be70f516aa6d2cb93921f260043f\": container with ID starting with ab6ad1c6ae3faeb9fe06cfac05f3427b2fc6be70f516aa6d2cb93921f260043f not found: ID does not exist" containerID="ab6ad1c6ae3faeb9fe06cfac05f3427b2fc6be70f516aa6d2cb93921f260043f"
Oct 29 23:11:13.883996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.883922    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6ad1c6ae3faeb9fe06cfac05f3427b2fc6be70f516aa6d2cb93921f260043f"} err="failed to get container status \"ab6ad1c6ae3faeb9fe06cfac05f3427b2fc6be70f516aa6d2cb93921f260043f\": rpc error: code = NotFound desc = could not find container \"ab6ad1c6ae3faeb9fe06cfac05f3427b2fc6be70f516aa6d2cb93921f260043f\": container with ID starting with ab6ad1c6ae3faeb9fe06cfac05f3427b2fc6be70f516aa6d2cb93921f260043f not found: ID does not exist"
Oct 29 23:11:13.927191 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.927162    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"]
Oct 29 23:11:13.939039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:13.939011    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7af1a5-5c65d57f7-24b8n"]
Oct 29 23:11:14.224477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:14.224426    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:14.347813 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:14.347762    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:14.769122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:14.769081    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:14.778323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:14.778296    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6112e33d-bd66-4ebc-8fb0-08a595027411" path="/var/lib/kubelet/pods/6112e33d-bd66-4ebc-8fb0-08a595027411/volumes"
Oct 29 23:11:16.170741 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:16.170708    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:17.374449 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:11:17.374394    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:11:17.968469 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:17.968429    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:18.717027 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:18.716990    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:18.841342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:18.841312    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:19.168912 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:19.167379    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:19.872927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:19.872881    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 23:11:19.875606 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:19.875043    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:19.962115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:19.962059    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded"
Oct 29 23:11:20.516826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:20.515896    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:11:20.750826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:20.750234    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:11:20.795885 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:20.793566    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:11:20.799824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:20.799060    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 29 23:11:20.883828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:20.856876    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:11:20.883828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:20.877902    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 23:11:20.979827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:20.962886    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:11:21.013418 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:20.990827    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:11:21.119884 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:21.101299    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 23:11:21.119884 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:21.101358    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 23:11:21.382903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:21.366893    5428 patch_prober.go:29] interesting pod/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:11:21.382903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:21.367009    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd" podUID="ae1fd856-9cb0-4d65-bc25-49aefe12e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:11:21.757884 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:21.754032    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:11:21.799828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:21.774352    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 23:11:21.828540 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:21.796882    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:11:21.865964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:21.865910    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:11:21.866543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:21.800236    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:11:21.880254 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:21.879434    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:11:21.911952 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:21.911914    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 23:11:21.913026 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:21.912997    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 23:11:21.913500 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:21.913477    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 23:11:22.025840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:21.990252    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:11:22.028640 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:22.025721    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:11:22.030567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:22.029625    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 23:11:22.535290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:22.533868    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 23:11:22.539553 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:22.535973    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 23:11:22.595665 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:22.595635    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 23:11:22.650848 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:22.640618    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 23:11:22.728994 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:22.728959    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 23:11:22.731086 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:22.731056    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 23:11:22.731239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:22.731176    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 23:11:25.414694 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:25.414662    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:27.070254 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:27.070219    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:27.346274 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:27.346189    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:29.044008 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:29.043972    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:30.789509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:30.789474    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:31.441955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:31.441919    5428 generic.go:356] "Generic (PLEG): container finished" podID="854e3883-53e9-4c18-ac91-bba8f9d4987e" containerID="fa3a7376c623bfade6df0807bb3d645174ccd8ed0a5e783be3e71798832e4608" exitCode=0
Oct 29 23:11:31.442124 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:31.441990    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w" event={"ID":"854e3883-53e9-4c18-ac91-bba8f9d4987e","Type":"ContainerDied","Data":"fa3a7376c623bfade6df0807bb3d645174ccd8ed0a5e783be3e71798832e4608"}
Oct 29 23:11:31.442422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:31.442406    5428 scope.go:122] "RemoveContainer" containerID="fa3a7376c623bfade6df0807bb3d645174ccd8ed0a5e783be3e71798832e4608"
Oct 29 23:11:31.444004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:31.443978    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:11:31.444075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:31.444047    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:11:31.444112 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:31.444078    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:11:32.450421 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:32.450388    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w" event={"ID":"854e3883-53e9-4c18-ac91-bba8f9d4987e","Type":"ContainerStarted","Data":"f43b76168bc8e8a51e0f255edcc1333a243118c119746fda5c66222637254321"}
Oct 29 23:11:32.454965 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:32.454936    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:33.242027 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:33.241989    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:11:33.242270 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:33.242227    5428 patch_prober.go:29] interesting pod/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 29 23:11:33.242366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:33.242282    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w" podUID="854e3883-53e9-4c18-ac91-bba8f9d4987e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 29 23:11:33.895876 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:33.888618    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"]
Oct 29 23:11:33.896672 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:33.896637    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j" podUID="7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26" containerName="maas-api" containerID="cri-o://6a3339763001bdcada2bb9e561cf5d0853de8352b7deaa979ff59dd46996355f" gracePeriod=30
Oct 29 23:11:33.930569 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:33.930541    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:34.113920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.113842    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:34.224470 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.222397    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:34.241908 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.241869    5428 patch_prober.go:29] interesting pod/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 29 23:11:34.242068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.241935    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w" podUID="854e3883-53e9-4c18-ac91-bba8f9d4987e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 29 23:11:34.480913 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.477454    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:11:34.512882 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.512851    5428 generic.go:356] "Generic (PLEG): container finished" podID="7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26" containerID="6a3339763001bdcada2bb9e561cf5d0853de8352b7deaa979ff59dd46996355f" exitCode=0
Oct 29 23:11:34.513096 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.512974    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"
Oct 29 23:11:34.513308 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.513011    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j" event={"ID":"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26","Type":"ContainerDied","Data":"6a3339763001bdcada2bb9e561cf5d0853de8352b7deaa979ff59dd46996355f"}
Oct 29 23:11:34.513470 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.513456    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j" event={"ID":"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26","Type":"ContainerDied","Data":"34910c0ebf419d842faacf07cfa2a47c818199aa6b974353c17e1c671c4e435c"}
Oct 29 23:11:34.513585 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.513576    5428 scope.go:122] "RemoveContainer" containerID="6a3339763001bdcada2bb9e561cf5d0853de8352b7deaa979ff59dd46996355f"
Oct 29 23:11:34.529569 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.529538    5428 scope.go:122] "RemoveContainer" containerID="6a3339763001bdcada2bb9e561cf5d0853de8352b7deaa979ff59dd46996355f"
Oct 29 23:11:34.530688 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:11:34.530658    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a3339763001bdcada2bb9e561cf5d0853de8352b7deaa979ff59dd46996355f\": container with ID starting with 6a3339763001bdcada2bb9e561cf5d0853de8352b7deaa979ff59dd46996355f not found: ID does not exist" containerID="6a3339763001bdcada2bb9e561cf5d0853de8352b7deaa979ff59dd46996355f"
Oct 29 23:11:34.531282 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.530700    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a3339763001bdcada2bb9e561cf5d0853de8352b7deaa979ff59dd46996355f"} err="failed to get container status \"6a3339763001bdcada2bb9e561cf5d0853de8352b7deaa979ff59dd46996355f\": rpc error: code = NotFound desc = could not find container \"6a3339763001bdcada2bb9e561cf5d0853de8352b7deaa979ff59dd46996355f\": container with ID starting with 6a3339763001bdcada2bb9e561cf5d0853de8352b7deaa979ff59dd46996355f not found: ID does not exist"
Oct 29 23:11:34.647450 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.647415    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-kube-api-access-5w5m5\" (UniqueName: \"kubernetes.io/projected/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-kube-api-access-5w5m5\") pod \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\" (UID: \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\") "
Oct 29 23:11:34.647649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.647499    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-maas-api-tls\") pod \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\" (UID: \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\") "
Oct 29 23:11:34.647649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.647586    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-openshift-service-ca\") pod \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\" (UID: \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\") "
Oct 29 23:11:34.647862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.647839    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-metrics-certs\") pod \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\" (UID: \"7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26\") "
Oct 29 23:11:34.648047 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.648007    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-openshift-service-ca" pod "7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26" (UID: "7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:34.648432 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.648407    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:11:34.649870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.649839    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-maas-api-tls" pod "7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26" (UID: "7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:11:34.650100 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.650078    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-kube-api-access-5w5m5" pod "7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26" (UID: "7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26"). InnerVolumeSpecName "kube-api-access-5w5m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:34.650249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.650226    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-metrics-certs" pod "7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26" (UID: "7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:11:34.750366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.750266    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w5m5\" (UniqueName: \"kubernetes.io/projected/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-kube-api-access-5w5m5\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:11:34.750366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.750317    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:11:34.750366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.750331    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:11:34.786463 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.786419    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:34.856258 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.856220    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"]
Oct 29 23:11:34.901275 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.901239    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-31d798-59dccf5fb9-p529j"]
Oct 29 23:11:34.942491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:34.942459    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:35.239830 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:35.236453    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:35.244186 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:35.244148    5428 patch_prober.go:29] interesting pod/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 29 23:11:35.244353 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:35.244209    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w" podUID="854e3883-53e9-4c18-ac91-bba8f9d4987e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 29 23:11:35.483933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:35.483195    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:36.242613 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:36.242580    5428 patch_prober.go:29] interesting pod/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 29 23:11:36.243029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:36.242658    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w" podUID="854e3883-53e9-4c18-ac91-bba8f9d4987e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 29 23:11:36.795031 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:36.794998    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26" path="/var/lib/kubelet/pods/7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26/volumes"
Oct 29 23:11:37.085658 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:37.085574    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:37.242885 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:37.242842    5428 patch_prober.go:29] interesting pod/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 29 23:11:37.243287 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:37.242918    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w" podUID="854e3883-53e9-4c18-ac91-bba8f9d4987e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 29 23:11:38.247113 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:38.247082    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:11:38.247509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:38.247375    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:11:38.248121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:38.248104    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:11:39.016504 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:39.016470    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:40.699273 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:40.699237    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:42.210077 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:42.210042    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:42.357451 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:42.357421    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:42.536556 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:42.536458    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:42.654824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:42.649601    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:44.128719 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:44.128675    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:44.536898 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:44.534524    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:44.662500 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:44.662461    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:44.789889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:44.789694    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:46.996393 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:46.996320    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:47.368002 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:11:47.367961    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:11:47.894690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:47.894655    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27"]
Oct 29 23:11:47.992291 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:47.992254    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:48.108785 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:48.108753    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:48.350559 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:48.350469    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:48.510936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:48.510844    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:49.266486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:49.266456    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:49.986634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:49.986594    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:50.103648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:50.103611    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:51.190563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:51.190534    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:52.895485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:52.895449    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27" podUID="9f7ba3a0-58c9-4bfb-a0d7-cf7786563277" containerName="payload-pre-processing" containerID="cri-o://3e4e4f5eb190572d2e17214968f776d2756779115e8f52e26de45d9aad9f7a90" gracePeriod=25
Oct 29 23:11:52.981795 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:52.981759    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:53.209907 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:53.209885    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27"
Oct 29 23:11:53.239910 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:53.239882    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f7ba3a0-58c9-4bfb-a0d7-cf7786563277-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7ba3a0-58c9-4bfb-a0d7-cf7786563277-plugins-config-volume\") pod \"9f7ba3a0-58c9-4bfb-a0d7-cf7786563277\" (UID: \"9f7ba3a0-58c9-4bfb-a0d7-cf7786563277\") "
Oct 29 23:11:53.240055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:53.240022    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f7ba3a0-58c9-4bfb-a0d7-cf7786563277-kube-api-access-96hgj\" (UniqueName: \"kubernetes.io/projected/9f7ba3a0-58c9-4bfb-a0d7-cf7786563277-kube-api-access-96hgj\") pod \"9f7ba3a0-58c9-4bfb-a0d7-cf7786563277\" (UID: \"9f7ba3a0-58c9-4bfb-a0d7-cf7786563277\") "
Oct 29 23:11:53.240278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:53.240254    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7ba3a0-58c9-4bfb-a0d7-cf7786563277-plugins-config-volume" pod "9f7ba3a0-58c9-4bfb-a0d7-cf7786563277" (UID: "9f7ba3a0-58c9-4bfb-a0d7-cf7786563277"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:53.240672 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:53.240656    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7ba3a0-58c9-4bfb-a0d7-cf7786563277-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:11:53.242051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:53.242033    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7ba3a0-58c9-4bfb-a0d7-cf7786563277-kube-api-access-96hgj" pod "9f7ba3a0-58c9-4bfb-a0d7-cf7786563277" (UID: "9f7ba3a0-58c9-4bfb-a0d7-cf7786563277"). InnerVolumeSpecName "kube-api-access-96hgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:53.342355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:53.342324    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96hgj\" (UniqueName: \"kubernetes.io/projected/9f7ba3a0-58c9-4bfb-a0d7-cf7786563277-kube-api-access-96hgj\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:11:53.748544 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:53.748513    5428 generic.go:356] "Generic (PLEG): container finished" podID="9f7ba3a0-58c9-4bfb-a0d7-cf7786563277" containerID="3e4e4f5eb190572d2e17214968f776d2756779115e8f52e26de45d9aad9f7a90" exitCode=0
Oct 29 23:11:53.748744 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:53.748567    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27"
Oct 29 23:11:53.748744 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:53.748602    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27" event={"ID":"9f7ba3a0-58c9-4bfb-a0d7-cf7786563277","Type":"ContainerDied","Data":"3e4e4f5eb190572d2e17214968f776d2756779115e8f52e26de45d9aad9f7a90"}
Oct 29 23:11:53.748744 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:53.748660    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27" event={"ID":"9f7ba3a0-58c9-4bfb-a0d7-cf7786563277","Type":"ContainerDied","Data":"9490dbd28c50cb88c669e29298ac8e76f172f734ed3ae5e5dcc913e8a21d7248"}
Oct 29 23:11:53.748744 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:53.748690    5428 scope.go:122] "RemoveContainer" containerID="3e4e4f5eb190572d2e17214968f776d2756779115e8f52e26de45d9aad9f7a90"
Oct 29 23:11:53.759584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:53.759564    5428 scope.go:122] "RemoveContainer" containerID="3e4e4f5eb190572d2e17214968f776d2756779115e8f52e26de45d9aad9f7a90"
Oct 29 23:11:53.759845 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:11:53.759824    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4e4f5eb190572d2e17214968f776d2756779115e8f52e26de45d9aad9f7a90\": container with ID starting with 3e4e4f5eb190572d2e17214968f776d2756779115e8f52e26de45d9aad9f7a90 not found: ID does not exist" containerID="3e4e4f5eb190572d2e17214968f776d2756779115e8f52e26de45d9aad9f7a90"
Oct 29 23:11:53.759942 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:53.759851    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4e4f5eb190572d2e17214968f776d2756779115e8f52e26de45d9aad9f7a90"} err="failed to get container status \"3e4e4f5eb190572d2e17214968f776d2756779115e8f52e26de45d9aad9f7a90\": rpc error: code = NotFound desc = could not find container \"3e4e4f5eb190572d2e17214968f776d2756779115e8f52e26de45d9aad9f7a90\": container with ID starting with 3e4e4f5eb190572d2e17214968f776d2756779115e8f52e26de45d9aad9f7a90 not found: ID does not exist"
Oct 29 23:11:53.801743 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:53.801708    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27"]
Oct 29 23:11:53.819296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:53.819263    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f2c8d3-6c8d55dd47-ljx27"]
Oct 29 23:11:54.777214 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:54.777179    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f7ba3a0-58c9-4bfb-a0d7-cf7786563277" path="/var/lib/kubelet/pods/9f7ba3a0-58c9-4bfb-a0d7-cf7786563277/volumes"
Oct 29 23:11:54.831726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:54.831687    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:54.942851 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:54.942777    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:55.413115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:55.413076    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:55.630675 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:55.630638    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:55.940421 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:55.940393    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:56.963877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:56.963833    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:57.735867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:57.734250    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:58.909269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:58.909235    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:59.701821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:59.696652    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:59.845827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:11:59.842744    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:00.894826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:00.891133    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:01.031830 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:01.030883    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:01.561697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:01.559903    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:01.561697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:01.560299    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 23:12:01.645939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:01.645888    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:01.732941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:01.732893    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:01.737918 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:01.737878    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:01.898019 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:01.897964    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:01.900121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:01.900092    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 23:12:02.041842 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:02.031900    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:02.041842 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:02.032374    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 23:12:02.589701 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:02.589660    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:02.594132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:02.594103    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 23:12:02.657701 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:02.657672    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 23:12:02.730936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:02.730887    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:02.754204 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:02.753742    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:02.774846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:02.767296    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 23:12:02.831491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:02.804872    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:02.935297 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:02.907875    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:03.396821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.389753    5428 patch_prober.go:29] interesting pod/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:12:03.396821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.389912    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm" podUID="9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:03.796862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.731685    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:03.873923 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.873874    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:03.910440 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:12:03.905515    5428 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.097s"
Oct 29 23:12:03.910440 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.905639    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 23:12:03.938415 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.932934    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:03.938415 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.933046    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58"]
Oct 29 23:12:03.941115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.939382    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9f7ba3a0-58c9-4bfb-a0d7-cf7786563277" containerName="payload-pre-processing"
Oct 29 23:12:03.941115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.939404    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f7ba3a0-58c9-4bfb-a0d7-cf7786563277" containerName="payload-pre-processing"
Oct 29 23:12:03.941115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.939450    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b38d4301-782b-4d87-a7f3-3b6036648288" containerName="payload-processing"
Oct 29 23:12:03.941115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.939458    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b38d4301-782b-4d87-a7f3-3b6036648288" containerName="payload-processing"
Oct 29 23:12:03.941115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.939501    5428 cpu_manager.go:414] "containerMap: removing container" podUID="7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26" containerName="maas-api"
Oct 29 23:12:03.941115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.939509    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26" containerName="maas-api"
Oct 29 23:12:03.941115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.939531    5428 cpu_manager.go:414] "containerMap: removing container" podUID="6112e33d-bd66-4ebc-8fb0-08a595027411" containerName="maas-api"
Oct 29 23:12:03.941115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.939538    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6112e33d-bd66-4ebc-8fb0-08a595027411" containerName="maas-api"
Oct 29 23:12:03.941115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.939556    5428 cpu_manager.go:414] "containerMap: removing container" podUID="181b8c86-e1c1-4208-9c6b-8a249d58c2a8" containerName="payload-pre-processing"
Oct 29 23:12:03.941115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.939563    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181b8c86-e1c1-4208-9c6b-8a249d58c2a8" containerName="payload-pre-processing"
Oct 29 23:12:03.941115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.939707    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="181b8c86-e1c1-4208-9c6b-8a249d58c2a8" containerName="payload-pre-processing"
Oct 29 23:12:03.941115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.939718    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f7ba3a0-58c9-4bfb-a0d7-cf7786563277" containerName="payload-pre-processing"
Oct 29 23:12:03.941115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.939733    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b38d4301-782b-4d87-a7f3-3b6036648288" containerName="payload-processing"
Oct 29 23:12:03.941115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.939747    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f9a3e94-c03f-4ab1-8e78-e2a5aad2de26" containerName="maas-api"
Oct 29 23:12:03.941115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.939759    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6112e33d-bd66-4ebc-8fb0-08a595027411" containerName="maas-api"
Oct 29 23:12:03.944465 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.944435    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:03.949851 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.949777    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Oct 29 23:12:03.952796 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.952763    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58"
Oct 29 23:12:03.973837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.972660    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-26a2a4\""
Oct 29 23:12:03.973837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:03.973204    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-26a2a4-dockercfg-z548q\""
Oct 29 23:12:04.042104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.042073    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 23:12:04.092993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.092960    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58"]
Oct 29 23:12:04.094558 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.094523    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42hqp\" (UniqueName: \"kubernetes.io/projected/e990be60-57d3-49f5-961a-d40381905773-kube-api-access-42hqp\") pod \"payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58\" (UID: \"e990be60-57d3-49f5-961a-d40381905773\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58"
Oct 29 23:12:04.108497 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.107707    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e990be60-57d3-49f5-961a-d40381905773-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58\" (UID: \"e990be60-57d3-49f5-961a-d40381905773\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58"
Oct 29 23:12:04.262283 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.222034    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42hqp\" (UniqueName: \"kubernetes.io/projected/e990be60-57d3-49f5-961a-d40381905773-kube-api-access-42hqp\") pod \"payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58\" (UID: \"e990be60-57d3-49f5-961a-d40381905773\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58"
Oct 29 23:12:04.263351 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.262653    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e990be60-57d3-49f5-961a-d40381905773-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58\" (UID: \"e990be60-57d3-49f5-961a-d40381905773\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58"
Oct 29 23:12:04.264579 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.264552    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e990be60-57d3-49f5-961a-d40381905773-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58\" (UID: \"e990be60-57d3-49f5-961a-d40381905773\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58"
Oct 29 23:12:04.281679 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.276624    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 23:12:04.286166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.286138    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29"]
Oct 29 23:12:04.295918 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.295817    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42hqp\" (UniqueName: \"kubernetes.io/projected/e990be60-57d3-49f5-961a-d40381905773-kube-api-access-42hqp\") pod \"payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58\" (UID: \"e990be60-57d3-49f5-961a-d40381905773\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58"
Oct 29 23:12:04.315411 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.315384    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29"]
Oct 29 23:12:04.315605 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.315474    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29"
Oct 29 23:12:04.349826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.349782    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58"
Oct 29 23:12:04.364954 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.364923    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103e3adb-1b76-44e2-ba11-03d11fea6b70-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29\" (UID: \"103e3adb-1b76-44e2-ba11-03d11fea6b70\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29"
Oct 29 23:12:04.365437 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.365410    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkp2b\" (UniqueName: \"kubernetes.io/projected/103e3adb-1b76-44e2-ba11-03d11fea6b70-kube-api-access-qkp2b\") pod \"payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29\" (UID: \"103e3adb-1b76-44e2-ba11-03d11fea6b70\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29"
Oct 29 23:12:04.469492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.468388    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkp2b\" (UniqueName: \"kubernetes.io/projected/103e3adb-1b76-44e2-ba11-03d11fea6b70-kube-api-access-qkp2b\") pod \"payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29\" (UID: \"103e3adb-1b76-44e2-ba11-03d11fea6b70\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29"
Oct 29 23:12:04.469492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.468535    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103e3adb-1b76-44e2-ba11-03d11fea6b70-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29\" (UID: \"103e3adb-1b76-44e2-ba11-03d11fea6b70\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29"
Oct 29 23:12:04.469692 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.469525    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103e3adb-1b76-44e2-ba11-03d11fea6b70-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29\" (UID: \"103e3adb-1b76-44e2-ba11-03d11fea6b70\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29"
Oct 29 23:12:04.493377 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.493338    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkp2b\" (UniqueName: \"kubernetes.io/projected/103e3adb-1b76-44e2-ba11-03d11fea6b70-kube-api-access-qkp2b\") pod \"payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29\" (UID: \"103e3adb-1b76-44e2-ba11-03d11fea6b70\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29"
Oct 29 23:12:04.579762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.579645    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:04.647630 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.647591    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29"
Oct 29 23:12:04.695884 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.695834    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58"]
Oct 29 23:12:04.695884 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:12:04.695860    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode990be60_57d3_49f5_961a_d40381905773.slice/crio-190304d858967328457aeefc529135244a345da3ea7fc1bef2b515efc355ded8 WatchSource:0}: Error finding container 190304d858967328457aeefc529135244a345da3ea7fc1bef2b515efc355ded8: Status 404 returned error can't find the container with id 190304d858967328457aeefc529135244a345da3ea7fc1bef2b515efc355ded8
Oct 29 23:12:04.735362 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.735324    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:04.891138 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.891109    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:04.897063 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.896977    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58" event={"ID":"e990be60-57d3-49f5-961a-d40381905773","Type":"ContainerStarted","Data":"33238e783b51fee2b7ffb3496f342d8142d6b458fd743b2fc802e337710e0003"}
Oct 29 23:12:04.897989 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.897963    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58"
Oct 29 23:12:04.898229 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.898012    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58" event={"ID":"e990be60-57d3-49f5-961a-d40381905773","Type":"ContainerStarted","Data":"190304d858967328457aeefc529135244a345da3ea7fc1bef2b515efc355ded8"}
Oct 29 23:12:04.954446 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.954398    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58" podStartSLOduration=1.954381419 podStartE2EDuration="1.954381419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:04.951031353 +0000 UTC m=+1728.293980101" watchObservedRunningTime="2026-10-29 23:12:04.954381419 +0000 UTC m=+1728.297330163"
Oct 29 23:12:04.991766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:04.991743    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29"]
Oct 29 23:12:04.992901 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:12:04.992871    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103e3adb_1b76_44e2_ba11_03d11fea6b70.slice/crio-f4f0010911b5ada9dd4ddbc6b6579cffa3fd85946a450515ee16c088e37799d5 WatchSource:0}: Error finding container f4f0010911b5ada9dd4ddbc6b6579cffa3fd85946a450515ee16c088e37799d5: Status 404 returned error can't find the container with id f4f0010911b5ada9dd4ddbc6b6579cffa3fd85946a450515ee16c088e37799d5
Oct 29 23:12:05.028575 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:05.028546    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:05.906496 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:05.906449    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29" event={"ID":"103e3adb-1b76-44e2-ba11-03d11fea6b70","Type":"ContainerStarted","Data":"ffe14db32e335551daa2be51aec2bcd36e4f081028c37b0e9b0db0a7aff3eb00"}
Oct 29 23:12:05.906676 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:05.906521    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29" event={"ID":"103e3adb-1b76-44e2-ba11-03d11fea6b70","Type":"ContainerStarted","Data":"f4f0010911b5ada9dd4ddbc6b6579cffa3fd85946a450515ee16c088e37799d5"}
Oct 29 23:12:05.906772 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:05.906754    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29"
Oct 29 23:12:05.949242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:05.949180    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29" podStartSLOduration=2.94916628 podStartE2EDuration="2.94916628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:05.946753475 +0000 UTC m=+1729.289702221" watchObservedRunningTime="2026-10-29 23:12:05.94916628 +0000 UTC m=+1729.292115024"
Oct 29 23:12:07.799476 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:07.799434    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:09.213901 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.213865    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:09.270648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.270615    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"]
Oct 29 23:12:09.270865 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.270840    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv" podUID="412ce42d-92e1-4a65-9378-e78592256758" containerName="maas-api" containerID="cri-o://056e8d2f9c1a5a313a643160f7fe86741917fba52e552221ccbfeebc1545bcf5" gracePeriod=30
Oct 29 23:12:09.348387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.348354    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:09.582769 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.582729    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"
Oct 29 23:12:09.687614 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.682036    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/412ce42d-92e1-4a65-9378-e78592256758-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/412ce42d-92e1-4a65-9378-e78592256758-maas-api-tls\") pod \"412ce42d-92e1-4a65-9378-e78592256758\" (UID: \"412ce42d-92e1-4a65-9378-e78592256758\") "
Oct 29 23:12:09.687614 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.682100    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/412ce42d-92e1-4a65-9378-e78592256758-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/412ce42d-92e1-4a65-9378-e78592256758-openshift-service-ca\") pod \"412ce42d-92e1-4a65-9378-e78592256758\" (UID: \"412ce42d-92e1-4a65-9378-e78592256758\") "
Oct 29 23:12:09.687614 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.682295    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/412ce42d-92e1-4a65-9378-e78592256758-kube-api-access-brd5x\" (UniqueName: \"kubernetes.io/projected/412ce42d-92e1-4a65-9378-e78592256758-kube-api-access-brd5x\") pod \"412ce42d-92e1-4a65-9378-e78592256758\" (UID: \"412ce42d-92e1-4a65-9378-e78592256758\") "
Oct 29 23:12:09.687614 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.682332    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/412ce42d-92e1-4a65-9378-e78592256758-metrics-certs\" (UniqueName: \"kubernetes.io/secret/412ce42d-92e1-4a65-9378-e78592256758-metrics-certs\") pod \"412ce42d-92e1-4a65-9378-e78592256758\" (UID: \"412ce42d-92e1-4a65-9378-e78592256758\") "
Oct 29 23:12:09.688023 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.687867    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/412ce42d-92e1-4a65-9378-e78592256758-openshift-service-ca" pod "412ce42d-92e1-4a65-9378-e78592256758" (UID: "412ce42d-92e1-4a65-9378-e78592256758"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:12:09.723083 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.688572    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/412ce42d-92e1-4a65-9378-e78592256758-metrics-certs" pod "412ce42d-92e1-4a65-9378-e78592256758" (UID: "412ce42d-92e1-4a65-9378-e78592256758"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:12:09.726865 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.723914    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/412ce42d-92e1-4a65-9378-e78592256758-maas-api-tls" pod "412ce42d-92e1-4a65-9378-e78592256758" (UID: "412ce42d-92e1-4a65-9378-e78592256758"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:12:09.735457 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.734941    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412ce42d-92e1-4a65-9378-e78592256758-kube-api-access-brd5x" pod "412ce42d-92e1-4a65-9378-e78592256758" (UID: "412ce42d-92e1-4a65-9378-e78592256758"). InnerVolumeSpecName "kube-api-access-brd5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:12:09.785993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.785960    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/412ce42d-92e1-4a65-9378-e78592256758-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:12:09.785993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.786000    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/412ce42d-92e1-4a65-9378-e78592256758-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:12:09.786197 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.786012    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brd5x\" (UniqueName: \"kubernetes.io/projected/412ce42d-92e1-4a65-9378-e78592256758-kube-api-access-brd5x\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:12:09.786197 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.786021    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/412ce42d-92e1-4a65-9378-e78592256758-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:12:09.945012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.944970    5428 generic.go:356] "Generic (PLEG): container finished" podID="412ce42d-92e1-4a65-9378-e78592256758" containerID="056e8d2f9c1a5a313a643160f7fe86741917fba52e552221ccbfeebc1545bcf5" exitCode=0
Oct 29 23:12:09.945218 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.945130    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv" event={"ID":"412ce42d-92e1-4a65-9378-e78592256758","Type":"ContainerDied","Data":"056e8d2f9c1a5a313a643160f7fe86741917fba52e552221ccbfeebc1545bcf5"}
Oct 29 23:12:09.945278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.945216    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv" event={"ID":"412ce42d-92e1-4a65-9378-e78592256758","Type":"ContainerDied","Data":"814e861ca175d6c059c499ee987093b29229548b0e0a038d6241d6cd28179d74"}
Oct 29 23:12:09.945278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.945247    5428 scope.go:122] "RemoveContainer" containerID="056e8d2f9c1a5a313a643160f7fe86741917fba52e552221ccbfeebc1545bcf5"
Oct 29 23:12:09.945442 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.945426    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"
Oct 29 23:12:09.967543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.967513    5428 scope.go:122] "RemoveContainer" containerID="056e8d2f9c1a5a313a643160f7fe86741917fba52e552221ccbfeebc1545bcf5"
Oct 29 23:12:09.969464 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:12:09.969433    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"056e8d2f9c1a5a313a643160f7fe86741917fba52e552221ccbfeebc1545bcf5\": container with ID starting with 056e8d2f9c1a5a313a643160f7fe86741917fba52e552221ccbfeebc1545bcf5 not found: ID does not exist" containerID="056e8d2f9c1a5a313a643160f7fe86741917fba52e552221ccbfeebc1545bcf5"
Oct 29 23:12:09.969574 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:09.969473    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"056e8d2f9c1a5a313a643160f7fe86741917fba52e552221ccbfeebc1545bcf5"} err="failed to get container status \"056e8d2f9c1a5a313a643160f7fe86741917fba52e552221ccbfeebc1545bcf5\": rpc error: code = NotFound desc = could not find container \"056e8d2f9c1a5a313a643160f7fe86741917fba52e552221ccbfeebc1545bcf5\": container with ID starting with 056e8d2f9c1a5a313a643160f7fe86741917fba52e552221ccbfeebc1545bcf5 not found: ID does not exist"
Oct 29 23:12:10.018393 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:10.018353    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"]
Oct 29 23:12:10.034229 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:10.034193    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f2c8d3-6974df64b-pl6jv"]
Oct 29 23:12:10.061921 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:10.061876    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:10.330709 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:10.330618    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:10.503064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:10.503029    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:10.634282 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:10.634249    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:10.760084 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:10.760053    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:10.780523 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:10.780485    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="412ce42d-92e1-4a65-9378-e78592256758" path="/var/lib/kubelet/pods/412ce42d-92e1-4a65-9378-e78592256758/volumes"
Oct 29 23:12:11.125089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:11.125060    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:12.711068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:12.711037    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:13.712138 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:13.712109    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:13.833364 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:13.833330    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:15.069400 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.069367    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:15.399921 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.399883    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:15.546868 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.546786    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:15.550074 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.550049    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"]
Oct 29 23:12:15.550765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.550749    5428 cpu_manager.go:414] "containerMap: removing container" podUID="412ce42d-92e1-4a65-9378-e78592256758" containerName="maas-api"
Oct 29 23:12:15.550830 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.550767    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="412ce42d-92e1-4a65-9378-e78592256758" containerName="maas-api"
Oct 29 23:12:15.550916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.550906    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="412ce42d-92e1-4a65-9378-e78592256758" containerName="maas-api"
Oct 29 23:12:15.554403 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.554385    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"
Oct 29 23:12:15.581850 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.581825    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-b73a75\""
Oct 29 23:12:15.582006 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.581931    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-b73a75-dockercfg-8t6gl\""
Oct 29 23:12:15.588158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.588131    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"]
Oct 29 23:12:15.643823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.638742    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9553cf74-cf03-4ad2-8241-d82087a823f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd\" (UID: \"9553cf74-cf03-4ad2-8241-d82087a823f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"
Oct 29 23:12:15.643823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.639028    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bznkp\" (UniqueName: \"kubernetes.io/projected/9553cf74-cf03-4ad2-8241-d82087a823f9-kube-api-access-bznkp\") pod \"payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd\" (UID: \"9553cf74-cf03-4ad2-8241-d82087a823f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"
Oct 29 23:12:15.754971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.738667    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"]
Oct 29 23:12:15.758783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.756131    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"
Oct 29 23:12:15.758783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.756907    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bznkp\" (UniqueName: \"kubernetes.io/projected/9553cf74-cf03-4ad2-8241-d82087a823f9-kube-api-access-bznkp\") pod \"payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd\" (UID: \"9553cf74-cf03-4ad2-8241-d82087a823f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"
Oct 29 23:12:15.758783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.757040    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9553cf74-cf03-4ad2-8241-d82087a823f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd\" (UID: \"9553cf74-cf03-4ad2-8241-d82087a823f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"
Oct 29 23:12:15.770361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.770315    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9553cf74-cf03-4ad2-8241-d82087a823f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd\" (UID: \"9553cf74-cf03-4ad2-8241-d82087a823f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"
Oct 29 23:12:15.781683 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.780646    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"]
Oct 29 23:12:15.800847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.792329    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bznkp\" (UniqueName: \"kubernetes.io/projected/9553cf74-cf03-4ad2-8241-d82087a823f9-kube-api-access-bznkp\") pod \"payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd\" (UID: \"9553cf74-cf03-4ad2-8241-d82087a823f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"
Oct 29 23:12:15.858209 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.858161    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtkcl\" (UniqueName: \"kubernetes.io/projected/9f1310de-26bc-46b9-86d6-10c8d1d49a93-kube-api-access-dtkcl\") pod \"payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj\" (UID: \"9f1310de-26bc-46b9-86d6-10c8d1d49a93\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"
Oct 29 23:12:15.858442 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.858381    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f1310de-26bc-46b9-86d6-10c8d1d49a93-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj\" (UID: \"9f1310de-26bc-46b9-86d6-10c8d1d49a93\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"
Oct 29 23:12:15.868233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.868204    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"
Oct 29 23:12:15.909840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.909796    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58"
Oct 29 23:12:15.965639 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.965469    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f1310de-26bc-46b9-86d6-10c8d1d49a93-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj\" (UID: \"9f1310de-26bc-46b9-86d6-10c8d1d49a93\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"
Oct 29 23:12:15.965794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.965646    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtkcl\" (UniqueName: \"kubernetes.io/projected/9f1310de-26bc-46b9-86d6-10c8d1d49a93-kube-api-access-dtkcl\") pod \"payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj\" (UID: \"9f1310de-26bc-46b9-86d6-10c8d1d49a93\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"
Oct 29 23:12:15.966759 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.966734    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f1310de-26bc-46b9-86d6-10c8d1d49a93-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj\" (UID: \"9f1310de-26bc-46b9-86d6-10c8d1d49a93\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"
Oct 29 23:12:15.989795 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:15.989759    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtkcl\" (UniqueName: \"kubernetes.io/projected/9f1310de-26bc-46b9-86d6-10c8d1d49a93-kube-api-access-dtkcl\") pod \"payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj\" (UID: \"9f1310de-26bc-46b9-86d6-10c8d1d49a93\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"
Oct 29 23:12:16.091215 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:16.091135    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"
Oct 29 23:12:16.191198 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:16.190173    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"]
Oct 29 23:12:16.403649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:16.403627    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"]
Oct 29 23:12:16.405748 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:12:16.405719    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1310de_26bc_46b9_86d6_10c8d1d49a93.slice/crio-f75c253bae1a18b475367f0e5677bc2c7aefa72f95415b328af1c6bacfd48efe WatchSource:0}: Error finding container f75c253bae1a18b475367f0e5677bc2c7aefa72f95415b328af1c6bacfd48efe: Status 404 returned error can't find the container with id f75c253bae1a18b475367f0e5677bc2c7aefa72f95415b328af1c6bacfd48efe
Oct 29 23:12:16.917618 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:16.917589    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29"
Oct 29 23:12:17.005507 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.005441    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd" event={"ID":"9553cf74-cf03-4ad2-8241-d82087a823f9","Type":"ContainerStarted","Data":"9473cc06c02a0b13020b16e281c4f2215afbf64e24a22ab3e9c07263362db820"}
Oct 29 23:12:17.005746 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.005519    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd" event={"ID":"9553cf74-cf03-4ad2-8241-d82087a823f9","Type":"ContainerStarted","Data":"f93841df843cba48f8c28b7339d44a84bc0406ae2fb7b4c9df0b683e42e149a5"}
Oct 29 23:12:17.005746 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.005574    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"
Oct 29 23:12:17.007724 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.007696    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj" event={"ID":"9f1310de-26bc-46b9-86d6-10c8d1d49a93","Type":"ContainerStarted","Data":"cbe4b6348319836c795de7c18c0186ebf073f26d8ab851f7fb7a09cb8a776c82"}
Oct 29 23:12:17.007851 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.007746    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj" event={"ID":"9f1310de-26bc-46b9-86d6-10c8d1d49a93","Type":"ContainerStarted","Data":"f75c253bae1a18b475367f0e5677bc2c7aefa72f95415b328af1c6bacfd48efe"}
Oct 29 23:12:17.007895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.007856    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"
Oct 29 23:12:17.030123 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.030097    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"]
Oct 29 23:12:17.034703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.034683    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"
Oct 29 23:12:17.047632 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.047588    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd" podStartSLOduration=2.047576637 podStartE2EDuration="2.047576637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:17.045665548 +0000 UTC m=+1740.388614294" watchObservedRunningTime="2026-10-29 23:12:17.047576637 +0000 UTC m=+1740.390525381"
Oct 29 23:12:17.064625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.062077    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"]
Oct 29 23:12:17.095703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.095636    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj" podStartSLOduration=2.095617354 podStartE2EDuration="2.095617354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:17.092662315 +0000 UTC m=+1740.435611062" watchObservedRunningTime="2026-10-29 23:12:17.095617354 +0000 UTC m=+1740.438566102"
Oct 29 23:12:17.126207 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.126179    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-96cd49-serving-cert\""
Oct 29 23:12:17.139974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.139944    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:17.182241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.182160    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtb9t\" (UniqueName: \"kubernetes.io/projected/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-kube-api-access-xtb9t\") pod \"maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg\" (UID: \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"
Oct 29 23:12:17.182241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.182199    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg\" (UID: \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"
Oct 29 23:12:17.182241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.182234    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-maas-api-tls\") pod \"maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg\" (UID: \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"
Oct 29 23:12:17.182439 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.182319    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-metrics-certs\") pod \"maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg\" (UID: \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"
Oct 29 23:12:17.273083 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.273044    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:17.283844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.283784    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtb9t\" (UniqueName: \"kubernetes.io/projected/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-kube-api-access-xtb9t\") pod \"maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg\" (UID: \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"
Oct 29 23:12:17.284011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.283939    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg\" (UID: \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"
Oct 29 23:12:17.284011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.283991    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-maas-api-tls\") pod \"maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg\" (UID: \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"
Oct 29 23:12:17.284151 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.284018    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-metrics-certs\") pod \"maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg\" (UID: \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"
Oct 29 23:12:17.284733 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.284703    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg\" (UID: \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"
Oct 29 23:12:17.287391 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.287364    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-metrics-certs\") pod \"maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg\" (UID: \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"
Oct 29 23:12:17.287539 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.287520    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-maas-api-tls\") pod \"maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg\" (UID: \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"
Oct 29 23:12:17.307560 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.307535    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtb9t\" (UniqueName: \"kubernetes.io/projected/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-kube-api-access-xtb9t\") pod \"maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg\" (UID: \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"
Oct 29 23:12:17.350086 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.350051    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"
Oct 29 23:12:17.377329 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:12:17.377293    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:12:17.674388 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.674357    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"]
Oct 29 23:12:17.675758 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:12:17.675731    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b7e549_f379_4461_ad77_c3ce9c1d00fc.slice/crio-0dbd72b558579f10061ec978466b4e1869bfed63d435245765d7c50ca89394f0 WatchSource:0}: Error finding container 0dbd72b558579f10061ec978466b4e1869bfed63d435245765d7c50ca89394f0: Status 404 returned error can't find the container with id 0dbd72b558579f10061ec978466b4e1869bfed63d435245765d7c50ca89394f0
Oct 29 23:12:17.728077 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.728049    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:17.850939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:17.850908    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:18.016761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:18.016653    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg" event={"ID":"a0b7e549-f379-4461-ad77-c3ce9c1d00fc","Type":"ContainerStarted","Data":"0dbd72b558579f10061ec978466b4e1869bfed63d435245765d7c50ca89394f0"}
Oct 29 23:12:18.097936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:18.097900    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:19.028301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:19.028263    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg" event={"ID":"a0b7e549-f379-4461-ad77-c3ce9c1d00fc","Type":"ContainerStarted","Data":"f06a9e3943737290e05fcf37a314c7aa950315f429247d610a4e553f03bb9178"}
Oct 29 23:12:19.028509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:19.028462    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"
Oct 29 23:12:19.070274 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:19.070210    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg" podStartSLOduration=1.546073595 podStartE2EDuration="2.070196517s" totalImagesPullingTime="524.122922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:19.069134784 +0000 UTC m=+1742.412083552" watchObservedRunningTime="2026-10-29 23:12:19.070196517 +0000 UTC m=+1742.413145259"
Oct 29 23:12:19.327393 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:19.327270    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:20.463507 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:20.463473    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:20.713590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:20.713511    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:20.939440 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:20.939403    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:21.105875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:21.096776    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:21.222691 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:21.222654    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:22.514569 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.514534    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"]
Oct 29 23:12:22.519334 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.519296    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.551770 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.550968    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"]
Oct 29 23:12:22.674731 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.674695    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.674955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.674741    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l9js\" (UniqueName: \"kubernetes.io/projected/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-kube-api-access-5l9js\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.674955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.674828    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.674955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.674908    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.675134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.675037    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.675134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.675070    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.675134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.675110    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.675294 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.675177    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.675294 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.675206    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.776855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.776755    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.776855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.776790    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.777091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.776885    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.777091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.776915    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l9js\" (UniqueName: \"kubernetes.io/projected/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-kube-api-access-5l9js\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.777091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.776984    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.777091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.777071    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.777309 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.777211    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.777309 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.777243    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.777309 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.777279    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.777470 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.777285    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.777470 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.777391    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.777582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.777542    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.777582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.777572    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.777685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.777584    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.779465 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.779436    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.779584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.779563    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.799958 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.799933    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.800070 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.799994    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l9js\" (UniqueName: \"kubernetes.io/projected/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-kube-api-access-5l9js\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:22.839746 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:22.839724    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:23.017315 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:23.017283    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:23.166762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:23.166730    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"]
Oct 29 23:12:23.167464 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:12:23.167437    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c4718e_8b21_4bee_a55c_22c06f27cd3a.slice/crio-3040bbf32a61d643a4a240d9828a561a3a7eebc059181f35985bd9bd9a57cf65 WatchSource:0}: Error finding container 3040bbf32a61d643a4a240d9828a561a3a7eebc059181f35985bd9bd9a57cf65: Status 404 returned error can't find the container with id 3040bbf32a61d643a4a240d9828a561a3a7eebc059181f35985bd9bd9a57cf65
Oct 29 23:12:23.170739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:23.170702    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:12:23.170872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:23.170778    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:12:23.170872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:23.170826    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:12:24.114208 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:24.114176    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4" event={"ID":"b2c4718e-8b21-4bee-a55c-22c06f27cd3a","Type":"ContainerStarted","Data":"7d02d9d7c0a6573f829cb5641e7834346f6076497f602fde1ab519d04b5b96b8"}
Oct 29 23:12:24.114621 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:24.114244    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4" event={"ID":"b2c4718e-8b21-4bee-a55c-22c06f27cd3a","Type":"ContainerStarted","Data":"3040bbf32a61d643a4a240d9828a561a3a7eebc059181f35985bd9bd9a57cf65"}
Oct 29 23:12:24.219631 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:24.219574    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4" podStartSLOduration=2.219560549 podStartE2EDuration="2.219560549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:24.216501593 +0000 UTC m=+1747.559450349" watchObservedRunningTime="2026-10-29 23:12:24.219560549 +0000 UTC m=+1747.562509293"
Oct 29 23:12:24.840747 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:24.840716    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:24.842562 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:24.842534    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 29 23:12:24.842690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:24.842586    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4" podUID="b2c4718e-8b21-4bee-a55c-22c06f27cd3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 29 23:12:24.941659 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:24.941630    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:25.041344 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:25.041316    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"
Oct 29 23:12:25.092562 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:25.092461    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:25.245474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:25.240255    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:25.840417 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:25.840381    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 29 23:12:25.840600 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:25.840437    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4" podUID="b2c4718e-8b21-4bee-a55c-22c06f27cd3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 29 23:12:26.685617 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:26.685577    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:26.795820 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:26.795771    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:26.840814 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:26.840769    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 29 23:12:26.840970 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:26.840833    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4" podUID="b2c4718e-8b21-4bee-a55c-22c06f27cd3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 29 23:12:26.891001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:26.890966    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58"]
Oct 29 23:12:26.896113 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:26.896086    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58"
Oct 29 23:12:26.897879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:26.897851    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58"]
Oct 29 23:12:26.912186 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:26.912161    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-b4b5e3\""
Oct 29 23:12:26.916860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:26.916834    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-b4b5e3-dockercfg-fkt2n\""
Oct 29 23:12:27.081585 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.081482    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925"]
Oct 29 23:12:27.124743 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.124186    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdzgr\" (UniqueName: \"kubernetes.io/projected/a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc-kube-api-access-jdzgr\") pod \"payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58\" (UID: \"a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58"
Oct 29 23:12:27.125317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.125233    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58\" (UID: \"a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58"
Oct 29 23:12:27.211321 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.211286    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925"]
Oct 29 23:12:27.211482 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.211412    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925"
Oct 29 23:12:27.226912 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.226883    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58\" (UID: \"a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58"
Oct 29 23:12:27.227059 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.227043    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdzgr\" (UniqueName: \"kubernetes.io/projected/a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc-kube-api-access-jdzgr\") pod \"payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58\" (UID: \"a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58"
Oct 29 23:12:27.227927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.227892    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58\" (UID: \"a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58"
Oct 29 23:12:27.250296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.250270    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdzgr\" (UniqueName: \"kubernetes.io/projected/a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc-kube-api-access-jdzgr\") pod \"payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58\" (UID: \"a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58"
Oct 29 23:12:27.328920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.328880    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fa0a50f-14ae-4ba8-a482-f7007672963c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925\" (UID: \"8fa0a50f-14ae-4ba8-a482-f7007672963c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925"
Oct 29 23:12:27.329094 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.328927    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prbdn\" (UniqueName: \"kubernetes.io/projected/8fa0a50f-14ae-4ba8-a482-f7007672963c-kube-api-access-prbdn\") pod \"payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925\" (UID: \"8fa0a50f-14ae-4ba8-a482-f7007672963c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925"
Oct 29 23:12:27.431022 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.430985    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fa0a50f-14ae-4ba8-a482-f7007672963c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925\" (UID: \"8fa0a50f-14ae-4ba8-a482-f7007672963c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925"
Oct 29 23:12:27.431022 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.431025    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prbdn\" (UniqueName: \"kubernetes.io/projected/8fa0a50f-14ae-4ba8-a482-f7007672963c-kube-api-access-prbdn\") pod \"payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925\" (UID: \"8fa0a50f-14ae-4ba8-a482-f7007672963c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925"
Oct 29 23:12:27.431687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.431666    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fa0a50f-14ae-4ba8-a482-f7007672963c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925\" (UID: \"8fa0a50f-14ae-4ba8-a482-f7007672963c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925"
Oct 29 23:12:27.456711 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.456681    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prbdn\" (UniqueName: \"kubernetes.io/projected/8fa0a50f-14ae-4ba8-a482-f7007672963c-kube-api-access-prbdn\") pod \"payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925\" (UID: \"8fa0a50f-14ae-4ba8-a482-f7007672963c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925"
Oct 29 23:12:27.518095 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.518056    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58"
Oct 29 23:12:27.530711 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.530679    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925"
Oct 29 23:12:27.630464 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.630432    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:27.801927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.801830    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:27.845293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.845255    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 29 23:12:27.845494 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.845398    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4" podUID="b2c4718e-8b21-4bee-a55c-22c06f27cd3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 29 23:12:27.858114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.856276    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58"]
Oct 29 23:12:27.906051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:27.906019    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925"]
Oct 29 23:12:27.907156 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:12:27.907127    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa0a50f_14ae_4ba8_a482_f7007672963c.slice/crio-1816aebff5fbb4b194ca63eb75f6208d89c7cf2ee6b687da7c0c5c73f1f97fde WatchSource:0}: Error finding container 1816aebff5fbb4b194ca63eb75f6208d89c7cf2ee6b687da7c0c5c73f1f97fde: Status 404 returned error can't find the container with id 1816aebff5fbb4b194ca63eb75f6208d89c7cf2ee6b687da7c0c5c73f1f97fde
Oct 29 23:12:28.020729 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:28.020686    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"
Oct 29 23:12:28.021877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:28.021856    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"
Oct 29 23:12:28.225650 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:28.225613    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925" event={"ID":"8fa0a50f-14ae-4ba8-a482-f7007672963c","Type":"ContainerStarted","Data":"8d3514c2389c95ae6bd7d2c10383887bc7952d6b2e326e86f49e248a10ee38ab"}
Oct 29 23:12:28.225890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:28.225710    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925" event={"ID":"8fa0a50f-14ae-4ba8-a482-f7007672963c","Type":"ContainerStarted","Data":"1816aebff5fbb4b194ca63eb75f6208d89c7cf2ee6b687da7c0c5c73f1f97fde"}
Oct 29 23:12:28.225890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:28.225775    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925"
Oct 29 23:12:28.227393 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:28.227368    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58" event={"ID":"a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc","Type":"ContainerStarted","Data":"58ec4a565f3488512f709912ab88c9a8c13e36011b58c5ae4d16e6037c1ae057"}
Oct 29 23:12:28.227514 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:28.227418    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58" event={"ID":"a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc","Type":"ContainerStarted","Data":"4d95471683cf5cb55bbbc14866a51983b3da4cfa60f4969fe08b55c10c70d73a"}
Oct 29 23:12:28.227514 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:28.227446    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58"
Oct 29 23:12:28.247599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:28.247577    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:28.288038 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:28.287977    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925" podStartSLOduration=2.287958597 podStartE2EDuration="2.287958597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:28.287179266 +0000 UTC m=+1751.630128012" watchObservedRunningTime="2026-10-29 23:12:28.287958597 +0000 UTC m=+1751.630907343"
Oct 29 23:12:28.330110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:28.330054    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58" podStartSLOduration=2.330040917 podStartE2EDuration="2.330040917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:28.327843522 +0000 UTC m=+1751.670792268" watchObservedRunningTime="2026-10-29 23:12:28.330040917 +0000 UTC m=+1751.672989662"
Oct 29 23:12:28.840096 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:28.840066    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 29 23:12:28.840473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:28.840114    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4" podUID="b2c4718e-8b21-4bee-a55c-22c06f27cd3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 29 23:12:29.150326 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:29.150289    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb"]
Oct 29 23:12:29.179165 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:29.179126    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz"]
Oct 29 23:12:29.284759 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:29.284725    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:29.418556 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:29.418474    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:29.840387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:29.840298    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 29 23:12:29.840387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:29.840351    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4" podUID="b2c4718e-8b21-4bee-a55c-22c06f27cd3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 29 23:12:30.681307 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:30.681251    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:30.808905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:30.808832    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:30.858481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:30.858451    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:30.859004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:30.858983    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:30.860006 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:30.859973    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:12:30.985641 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:30.985568    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:31.245512 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:31.245424    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:32.805163 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:32.805116    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:33.578125 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:33.578092    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:34.166827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.162865    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb" podUID="e2def0aa-9f21-412f-ba03-50ccd743d3bb" containerName="payload-processing" containerID="cri-o://d82b04aa763d9757e0d063307dc850ec31896034e38209300b2db353bf245222" gracePeriod=25
Oct 29 23:12:34.180449 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.179872    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz" podUID="7b3b173c-cc70-45a8-a2c6-7f1321e611c3" containerName="payload-pre-processing" containerID="cri-o://60d98cdac4cc9b90765637dc42ab4c565106d0f1152a75664e8e0d1184dfb734" gracePeriod=25
Oct 29 23:12:34.207879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.187379    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:34.348789 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.348751    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:34.566373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.566340    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz"
Oct 29 23:12:34.643658 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.643617    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb"
Oct 29 23:12:34.673291 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.673259    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b3b173c-cc70-45a8-a2c6-7f1321e611c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b3b173c-cc70-45a8-a2c6-7f1321e611c3-plugins-config-volume\") pod \"7b3b173c-cc70-45a8-a2c6-7f1321e611c3\" (UID: \"7b3b173c-cc70-45a8-a2c6-7f1321e611c3\") "
Oct 29 23:12:34.673477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.673328    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b3b173c-cc70-45a8-a2c6-7f1321e611c3-kube-api-access-d7r27\" (UniqueName: \"kubernetes.io/projected/7b3b173c-cc70-45a8-a2c6-7f1321e611c3-kube-api-access-d7r27\") pod \"7b3b173c-cc70-45a8-a2c6-7f1321e611c3\" (UID: \"7b3b173c-cc70-45a8-a2c6-7f1321e611c3\") "
Oct 29 23:12:34.673672 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.673643    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b3b173c-cc70-45a8-a2c6-7f1321e611c3-plugins-config-volume" pod "7b3b173c-cc70-45a8-a2c6-7f1321e611c3" (UID: "7b3b173c-cc70-45a8-a2c6-7f1321e611c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:12:34.675479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.675455    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3b173c-cc70-45a8-a2c6-7f1321e611c3-kube-api-access-d7r27" pod "7b3b173c-cc70-45a8-a2c6-7f1321e611c3" (UID: "7b3b173c-cc70-45a8-a2c6-7f1321e611c3"). InnerVolumeSpecName "kube-api-access-d7r27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:12:34.774297 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.774203    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2def0aa-9f21-412f-ba03-50ccd743d3bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2def0aa-9f21-412f-ba03-50ccd743d3bb-plugins-config-volume\") pod \"e2def0aa-9f21-412f-ba03-50ccd743d3bb\" (UID: \"e2def0aa-9f21-412f-ba03-50ccd743d3bb\") "
Oct 29 23:12:34.774478 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.774310    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2def0aa-9f21-412f-ba03-50ccd743d3bb-kube-api-access-5fm9t\" (UniqueName: \"kubernetes.io/projected/e2def0aa-9f21-412f-ba03-50ccd743d3bb-kube-api-access-5fm9t\") pod \"e2def0aa-9f21-412f-ba03-50ccd743d3bb\" (UID: \"e2def0aa-9f21-412f-ba03-50ccd743d3bb\") "
Oct 29 23:12:34.774649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.774620    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2def0aa-9f21-412f-ba03-50ccd743d3bb-plugins-config-volume" pod "e2def0aa-9f21-412f-ba03-50ccd743d3bb" (UID: "e2def0aa-9f21-412f-ba03-50ccd743d3bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:12:34.774959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.774937    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2def0aa-9f21-412f-ba03-50ccd743d3bb-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:12:34.775687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.774964    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b3b173c-cc70-45a8-a2c6-7f1321e611c3-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:12:34.775687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.774978    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7r27\" (UniqueName: \"kubernetes.io/projected/7b3b173c-cc70-45a8-a2c6-7f1321e611c3-kube-api-access-d7r27\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:12:34.776705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.776681    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2def0aa-9f21-412f-ba03-50ccd743d3bb-kube-api-access-5fm9t" pod "e2def0aa-9f21-412f-ba03-50ccd743d3bb" (UID: "e2def0aa-9f21-412f-ba03-50ccd743d3bb"). InnerVolumeSpecName "kube-api-access-5fm9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:12:34.875917 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.875887    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fm9t\" (UniqueName: \"kubernetes.io/projected/e2def0aa-9f21-412f-ba03-50ccd743d3bb-kube-api-access-5fm9t\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:12:34.887339 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:34.887313    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:35.306734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:35.306700    5428 generic.go:356] "Generic (PLEG): container finished" podID="e2def0aa-9f21-412f-ba03-50ccd743d3bb" containerID="d82b04aa763d9757e0d063307dc850ec31896034e38209300b2db353bf245222" exitCode=0
Oct 29 23:12:35.307237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:35.306797    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb"
Oct 29 23:12:35.307237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:35.306846    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb" event={"ID":"e2def0aa-9f21-412f-ba03-50ccd743d3bb","Type":"ContainerDied","Data":"d82b04aa763d9757e0d063307dc850ec31896034e38209300b2db353bf245222"}
Oct 29 23:12:35.307237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:35.306907    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb" event={"ID":"e2def0aa-9f21-412f-ba03-50ccd743d3bb","Type":"ContainerDied","Data":"73a298b019a8509a83d390a5d23f1494200fd9ad8cf8b8658249023777518435"}
Oct 29 23:12:35.307237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:35.306940    5428 scope.go:122] "RemoveContainer" containerID="d82b04aa763d9757e0d063307dc850ec31896034e38209300b2db353bf245222"
Oct 29 23:12:35.308656 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:35.308636    5428 generic.go:356] "Generic (PLEG): container finished" podID="7b3b173c-cc70-45a8-a2c6-7f1321e611c3" containerID="60d98cdac4cc9b90765637dc42ab4c565106d0f1152a75664e8e0d1184dfb734" exitCode=0
Oct 29 23:12:35.308793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:35.308692    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz"
Oct 29 23:12:35.308793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:35.308765    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz" event={"ID":"7b3b173c-cc70-45a8-a2c6-7f1321e611c3","Type":"ContainerDied","Data":"60d98cdac4cc9b90765637dc42ab4c565106d0f1152a75664e8e0d1184dfb734"}
Oct 29 23:12:35.308949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:35.308850    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz" event={"ID":"7b3b173c-cc70-45a8-a2c6-7f1321e611c3","Type":"ContainerDied","Data":"a07ceec7156cfcbcd10778e848e03590829d3a2fca6a173716874c0978cb22d9"}
Oct 29 23:12:35.317855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:35.317836    5428 scope.go:122] "RemoveContainer" containerID="d82b04aa763d9757e0d063307dc850ec31896034e38209300b2db353bf245222"
Oct 29 23:12:35.318118 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:12:35.318101    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d82b04aa763d9757e0d063307dc850ec31896034e38209300b2db353bf245222\": container with ID starting with d82b04aa763d9757e0d063307dc850ec31896034e38209300b2db353bf245222 not found: ID does not exist" containerID="d82b04aa763d9757e0d063307dc850ec31896034e38209300b2db353bf245222"
Oct 29 23:12:35.318163 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:35.318126    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82b04aa763d9757e0d063307dc850ec31896034e38209300b2db353bf245222"} err="failed to get container status \"d82b04aa763d9757e0d063307dc850ec31896034e38209300b2db353bf245222\": rpc error: code = NotFound desc = could not find container \"d82b04aa763d9757e0d063307dc850ec31896034e38209300b2db353bf245222\": container with ID starting with d82b04aa763d9757e0d063307dc850ec31896034e38209300b2db353bf245222 not found: ID does not exist"
Oct 29 23:12:35.318163 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:35.318142    5428 scope.go:122] "RemoveContainer" containerID="60d98cdac4cc9b90765637dc42ab4c565106d0f1152a75664e8e0d1184dfb734"
Oct 29 23:12:35.327482 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:35.327455    5428 scope.go:122] "RemoveContainer" containerID="60d98cdac4cc9b90765637dc42ab4c565106d0f1152a75664e8e0d1184dfb734"
Oct 29 23:12:35.327775 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:12:35.327740    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d98cdac4cc9b90765637dc42ab4c565106d0f1152a75664e8e0d1184dfb734\": container with ID starting with 60d98cdac4cc9b90765637dc42ab4c565106d0f1152a75664e8e0d1184dfb734 not found: ID does not exist" containerID="60d98cdac4cc9b90765637dc42ab4c565106d0f1152a75664e8e0d1184dfb734"
Oct 29 23:12:35.327862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:35.327775    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d98cdac4cc9b90765637dc42ab4c565106d0f1152a75664e8e0d1184dfb734"} err="failed to get container status \"60d98cdac4cc9b90765637dc42ab4c565106d0f1152a75664e8e0d1184dfb734\": rpc error: code = NotFound desc = could not find container \"60d98cdac4cc9b90765637dc42ab4c565106d0f1152a75664e8e0d1184dfb734\": container with ID starting with 60d98cdac4cc9b90765637dc42ab4c565106d0f1152a75664e8e0d1184dfb734 not found: ID does not exist"
Oct 29 23:12:35.364017 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:35.363990    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz"]
Oct 29 23:12:35.372132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:35.372110    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-65ffcb-5b889d855f-gs5fz"]
Oct 29 23:12:35.403413 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:35.403385    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb"]
Oct 29 23:12:35.409112 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:35.409089    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-65ffcb-7fc67cc74-76nmb"]
Oct 29 23:12:36.654290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:36.654224    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:36.780427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:36.780393    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b3b173c-cc70-45a8-a2c6-7f1321e611c3" path="/var/lib/kubelet/pods/7b3b173c-cc70-45a8-a2c6-7f1321e611c3/volumes"
Oct 29 23:12:36.781016 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:36.780993    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2def0aa-9f21-412f-ba03-50ccd743d3bb" path="/var/lib/kubelet/pods/e2def0aa-9f21-412f-ba03-50ccd743d3bb/volumes"
Oct 29 23:12:37.541178 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.541141    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"]
Oct 29 23:12:37.541858 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.541841    5428 cpu_manager.go:414] "containerMap: removing container" podUID="7b3b173c-cc70-45a8-a2c6-7f1321e611c3" containerName="payload-pre-processing"
Oct 29 23:12:37.541934 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.541858    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b3b173c-cc70-45a8-a2c6-7f1321e611c3" containerName="payload-pre-processing"
Oct 29 23:12:37.541934 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.541885    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e2def0aa-9f21-412f-ba03-50ccd743d3bb" containerName="payload-processing"
Oct 29 23:12:37.541934 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.541890    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2def0aa-9f21-412f-ba03-50ccd743d3bb" containerName="payload-processing"
Oct 29 23:12:37.542103 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.542033    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b3b173c-cc70-45a8-a2c6-7f1321e611c3" containerName="payload-pre-processing"
Oct 29 23:12:37.542103 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.542055    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2def0aa-9f21-412f-ba03-50ccd743d3bb" containerName="payload-processing"
Oct 29 23:12:37.547445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.547426    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.581884 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.581859    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"]
Oct 29 23:12:37.615536 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.615502    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.615730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.615567    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.615730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.615616    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.615937 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.615773    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.615937 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.615897    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.616051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.615941    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.616051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.615981    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.616169 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.616116    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsw76\" (UniqueName: \"kubernetes.io/projected/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-kube-api-access-jsw76\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.616169 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.616158    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.717956 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.717923    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.718389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.717972    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.718389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.717996    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.718389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.718022    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.718389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.718066    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsw76\" (UniqueName: \"kubernetes.io/projected/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-kube-api-access-jsw76\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.718389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.718088    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.718389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.718193    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.718389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.718278    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.718389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.718304    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.718852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.718621    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-credential-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.718852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.718658    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-workload-certs\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.718852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.718720    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-data\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.718852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.718734    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-workload-socket\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.719108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.718927    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istiod-ca-cert\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.720515 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.720493    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-envoy\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.720620 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.720551    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-podinfo\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.740752 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.740726    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsw76\" (UniqueName: \"kubernetes.io/projected/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-kube-api-access-jsw76\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.740898 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.740877    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-token\") pod \"e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") " pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:37.861770 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:37.861695    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:38.191885 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.191851    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"]
Oct 29 23:12:38.193580 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:12:38.193554    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e3f78e3_9804_4b4f_8e9d_6e37b0351643.slice/crio-62c6b0743e3c6168b7e78ae66104885a239fbea1d5356a12850e047dd1043db7 WatchSource:0}: Error finding container 62c6b0743e3c6168b7e78ae66104885a239fbea1d5356a12850e047dd1043db7: Status 404 returned error can't find the container with id 62c6b0743e3c6168b7e78ae66104885a239fbea1d5356a12850e047dd1043db7
Oct 29 23:12:38.196327 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.196282    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:12:38.196433 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.196373    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:12:38.196433 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.196421    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:12:38.335526 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.335486    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj" event={"ID":"8e3f78e3-9804-4b4f-8e9d-6e37b0351643","Type":"ContainerStarted","Data":"053af7b4509390659618ec8cc86d2bdc37944940469e3a1574727290435d520c"}
Oct 29 23:12:38.335703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.335550    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj" event={"ID":"8e3f78e3-9804-4b4f-8e9d-6e37b0351643","Type":"ContainerStarted","Data":"62c6b0743e3c6168b7e78ae66104885a239fbea1d5356a12850e047dd1043db7"}
Oct 29 23:12:38.386078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.386012    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj" podStartSLOduration=1.385991709 podStartE2EDuration="1.385991709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:38.381263098 +0000 UTC m=+1761.724211844" watchObservedRunningTime="2026-10-29 23:12:38.385991709 +0000 UTC m=+1761.728940457"
Oct 29 23:12:38.399398 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.399362    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"]
Oct 29 23:12:38.404583 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.404558    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:12:38.421108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.421084    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-b73a75-serving-cert\""
Oct 29 23:12:38.436787 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.436753    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"]
Oct 29 23:12:38.439759 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.439612    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b288d408-943e-4b45-abc9-7d942e5eb3f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh\" (UID: \"b288d408-943e-4b45-abc9-7d942e5eb3f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:12:38.440013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.439990    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b288d408-943e-4b45-abc9-7d942e5eb3f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh\" (UID: \"b288d408-943e-4b45-abc9-7d942e5eb3f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:12:38.440118 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.440047    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b288d408-943e-4b45-abc9-7d942e5eb3f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh\" (UID: \"b288d408-943e-4b45-abc9-7d942e5eb3f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:12:38.440118 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.440120    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sptlh\" (UniqueName: \"kubernetes.io/projected/b288d408-943e-4b45-abc9-7d942e5eb3f2-kube-api-access-sptlh\") pod \"maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh\" (UID: \"b288d408-943e-4b45-abc9-7d942e5eb3f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:12:38.529445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.529368    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:38.543104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.543073    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b288d408-943e-4b45-abc9-7d942e5eb3f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh\" (UID: \"b288d408-943e-4b45-abc9-7d942e5eb3f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:12:38.543253 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.543225    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b288d408-943e-4b45-abc9-7d942e5eb3f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh\" (UID: \"b288d408-943e-4b45-abc9-7d942e5eb3f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:12:38.543344 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.543257    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b288d408-943e-4b45-abc9-7d942e5eb3f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh\" (UID: \"b288d408-943e-4b45-abc9-7d942e5eb3f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:12:38.543344 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.543299    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sptlh\" (UniqueName: \"kubernetes.io/projected/b288d408-943e-4b45-abc9-7d942e5eb3f2-kube-api-access-sptlh\") pod \"maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh\" (UID: \"b288d408-943e-4b45-abc9-7d942e5eb3f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:12:38.543538 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:12:38.543372    5428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-serving-cert: secret "maas-api-e2e-shared-a-w2-b73a75-serving-cert" not found
Oct 29 23:12:38.543538 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:12:38.543461    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b288d408-943e-4b45-abc9-7d942e5eb3f2-maas-api-tls podName:b288d408-943e-4b45-abc9-7d942e5eb3f2 nodeName:}" failed. No retries permitted until 2026-10-29 23:12:39.04344154 +0000 UTC m=+1762.386390267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b288d408-943e-4b45-abc9-7d942e5eb3f2-maas-api-tls") pod "maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh" (UID: "b288d408-943e-4b45-abc9-7d942e5eb3f2") : secret "maas-api-e2e-shared-a-w2-b73a75-serving-cert" not found
Oct 29 23:12:38.544029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.544006    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b288d408-943e-4b45-abc9-7d942e5eb3f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh\" (UID: \"b288d408-943e-4b45-abc9-7d942e5eb3f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:12:38.545662 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.545643    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b288d408-943e-4b45-abc9-7d942e5eb3f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh\" (UID: \"b288d408-943e-4b45-abc9-7d942e5eb3f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:12:38.568700 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.568676    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sptlh\" (UniqueName: \"kubernetes.io/projected/b288d408-943e-4b45-abc9-7d942e5eb3f2-kube-api-access-sptlh\") pod \"maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh\" (UID: \"b288d408-943e-4b45-abc9-7d942e5eb3f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:12:38.662478 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.662447    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:38.802881 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.801135    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:38.877821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:38.874894    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:39.056394 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:39.056290    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b288d408-943e-4b45-abc9-7d942e5eb3f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh\" (UID: \"b288d408-943e-4b45-abc9-7d942e5eb3f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:12:39.059144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:39.059112    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b288d408-943e-4b45-abc9-7d942e5eb3f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh\" (UID: \"b288d408-943e-4b45-abc9-7d942e5eb3f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:12:39.255707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:39.255679    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58"
Oct 29 23:12:39.255924 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:39.255745    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925"
Oct 29 23:12:39.324787 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:39.324690    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:12:39.712625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:39.712243    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"]
Oct 29 23:12:39.715816 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:12:39.715773    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb288d408_943e_4b45_abc9_7d942e5eb3f2.slice/crio-64e999f02ab4407c7e6b3fb2fbe722b3690d3d9aa19ff3aaa557e23f1c4cb2ba WatchSource:0}: Error finding container 64e999f02ab4407c7e6b3fb2fbe722b3690d3d9aa19ff3aaa557e23f1c4cb2ba: Status 404 returned error can't find the container with id 64e999f02ab4407c7e6b3fb2fbe722b3690d3d9aa19ff3aaa557e23f1c4cb2ba
Oct 29 23:12:39.885827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:39.879967    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 29 23:12:39.885827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:39.880037    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj" podUID="8e3f78e3-9804-4b4f-8e9d-6e37b0351643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 29 23:12:40.400316 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.400145    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"]
Oct 29 23:12:40.400533 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.400439    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss" podUID="9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d" containerName="maas-api" containerID="cri-o://2816a900341e1a9b2e673418e938bacdcfce49bfaa905fb9ae2cafe851a76964" gracePeriod=30
Oct 29 23:12:40.406507 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.402952    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:40.410379 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.410219    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh" event={"ID":"b288d408-943e-4b45-abc9-7d942e5eb3f2","Type":"ContainerStarted","Data":"8887ba9811f43b93a7be4876baf29c15697fde3b2c02dfc6057d3fe3f1b7725e"}
Oct 29 23:12:40.410784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.410767    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:12:40.410973 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.410956    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh" event={"ID":"b288d408-943e-4b45-abc9-7d942e5eb3f2","Type":"ContainerStarted","Data":"64e999f02ab4407c7e6b3fb2fbe722b3690d3d9aa19ff3aaa557e23f1c4cb2ba"}
Oct 29 23:12:40.518118 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.518062    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh" podStartSLOduration=2.104030367 podStartE2EDuration="2.518045926s" totalImagesPullingTime="414.015559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:40.513795104 +0000 UTC m=+1763.856743850" watchObservedRunningTime="2026-10-29 23:12:40.518045926 +0000 UTC m=+1763.860994671"
Oct 29 23:12:40.539700 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.539667    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:40.708911 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.700016    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:40.708911 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.708893    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp"]
Oct 29 23:12:40.715211 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.715180    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp"
Oct 29 23:12:40.715629 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.715603    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp"]
Oct 29 23:12:40.734179 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.733016    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-2126fb\""
Oct 29 23:12:40.734179 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.733686    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-2126fb-dockercfg-6cmwf\""
Oct 29 23:12:40.816718 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.814848    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv29r\" (UniqueName: \"kubernetes.io/projected/b73ad056-0a89-4c23-84b7-d0eceab57c0b-kube-api-access-hv29r\") pod \"payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp\" (UID: \"b73ad056-0a89-4c23-84b7-d0eceab57c0b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp"
Oct 29 23:12:40.816718 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.814909    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73ad056-0a89-4c23-84b7-d0eceab57c0b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp\" (UID: \"b73ad056-0a89-4c23-84b7-d0eceab57c0b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp"
Oct 29 23:12:40.821281 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.821231    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:40.875852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.873199    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 29 23:12:40.875852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.873259    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj" podUID="8e3f78e3-9804-4b4f-8e9d-6e37b0351643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 29 23:12:40.940825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.939977    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73ad056-0a89-4c23-84b7-d0eceab57c0b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp\" (UID: \"b73ad056-0a89-4c23-84b7-d0eceab57c0b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp"
Oct 29 23:12:40.940825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.940495    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv29r\" (UniqueName: \"kubernetes.io/projected/b73ad056-0a89-4c23-84b7-d0eceab57c0b-kube-api-access-hv29r\") pod \"payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp\" (UID: \"b73ad056-0a89-4c23-84b7-d0eceab57c0b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp"
Oct 29 23:12:40.977884 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:40.941711    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73ad056-0a89-4c23-84b7-d0eceab57c0b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp\" (UID: \"b73ad056-0a89-4c23-84b7-d0eceab57c0b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp"
Oct 29 23:12:41.080006 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.068439    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv29r\" (UniqueName: \"kubernetes.io/projected/b73ad056-0a89-4c23-84b7-d0eceab57c0b-kube-api-access-hv29r\") pod \"payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp\" (UID: \"b73ad056-0a89-4c23-84b7-d0eceab57c0b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp"
Oct 29 23:12:41.225047 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.223389    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:41.253265 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.252775    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"
Oct 29 23:12:41.349841 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.348907    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp"
Oct 29 23:12:41.435943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.405296    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:41.435943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.405963    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-maas-api-tls\") pod \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\" (UID: \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\") "
Oct 29 23:12:41.435943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.406098    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-kube-api-access-4pdl4\" (UniqueName: \"kubernetes.io/projected/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-kube-api-access-4pdl4\") pod \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\" (UID: \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\") "
Oct 29 23:12:41.435943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.406279    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-openshift-service-ca\") pod \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\" (UID: \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\") "
Oct 29 23:12:41.435943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.406410    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-metrics-certs\") pod \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\" (UID: \"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d\") "
Oct 29 23:12:41.435943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.407700    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-openshift-service-ca" pod "9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d" (UID: "9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:12:41.435943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.413105    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-kube-api-access-4pdl4" pod "9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d" (UID: "9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d"). InnerVolumeSpecName "kube-api-access-4pdl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:12:41.435943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.415206    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-maas-api-tls" pod "9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d" (UID: "9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:12:41.435943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.425579    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-metrics-certs" pod "9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d" (UID: "9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:12:41.435943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.431581    5428 generic.go:356] "Generic (PLEG): container finished" podID="9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d" containerID="2816a900341e1a9b2e673418e938bacdcfce49bfaa905fb9ae2cafe851a76964" exitCode=0
Oct 29 23:12:41.435943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.431903    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"
Oct 29 23:12:41.435943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.432073    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss" event={"ID":"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d","Type":"ContainerDied","Data":"2816a900341e1a9b2e673418e938bacdcfce49bfaa905fb9ae2cafe851a76964"}
Oct 29 23:12:41.435943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.432127    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss" event={"ID":"9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d","Type":"ContainerDied","Data":"375334d96e06042610406830fb5f0f52a4705e9100bd9f0be6ba51c215e5939f"}
Oct 29 23:12:41.435943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.432160    5428 scope.go:122] "RemoveContainer" containerID="2816a900341e1a9b2e673418e938bacdcfce49bfaa905fb9ae2cafe851a76964"
Oct 29 23:12:41.454530 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.454273    5428 scope.go:122] "RemoveContainer" containerID="2816a900341e1a9b2e673418e938bacdcfce49bfaa905fb9ae2cafe851a76964"
Oct 29 23:12:41.454889 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:12:41.454783    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2816a900341e1a9b2e673418e938bacdcfce49bfaa905fb9ae2cafe851a76964\": container with ID starting with 2816a900341e1a9b2e673418e938bacdcfce49bfaa905fb9ae2cafe851a76964 not found: ID does not exist" containerID="2816a900341e1a9b2e673418e938bacdcfce49bfaa905fb9ae2cafe851a76964"
Oct 29 23:12:41.454889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.454834    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2816a900341e1a9b2e673418e938bacdcfce49bfaa905fb9ae2cafe851a76964"} err="failed to get container status \"2816a900341e1a9b2e673418e938bacdcfce49bfaa905fb9ae2cafe851a76964\": rpc error: code = NotFound desc = could not find container \"2816a900341e1a9b2e673418e938bacdcfce49bfaa905fb9ae2cafe851a76964\": container with ID starting with 2816a900341e1a9b2e673418e938bacdcfce49bfaa905fb9ae2cafe851a76964 not found: ID does not exist"
Oct 29 23:12:41.513491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.510848    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"]
Oct 29 23:12:41.513491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.511787    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pdl4\" (UniqueName: \"kubernetes.io/projected/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-kube-api-access-4pdl4\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:12:41.513491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.511844    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:12:41.513491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.511861    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:12:41.513491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.511875    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:12:41.524356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.519989    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-618d6c-6fd589b547-rjlss"]
Oct 29 23:12:41.851958 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.845777    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp"]
Oct 29 23:12:41.905204 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.897765    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 29 23:12:41.905204 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:41.897857    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj" podUID="8e3f78e3-9804-4b4f-8e9d-6e37b0351643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 29 23:12:42.111080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:42.108961    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:42.254863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:42.252945    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:42.473056 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:42.472902    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp" event={"ID":"b73ad056-0a89-4c23-84b7-d0eceab57c0b","Type":"ContainerStarted","Data":"0434f934ca2361be0ebe460a9fff4d492a32c14d1799824502ae1af38654a2a0"}
Oct 29 23:12:42.473056 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:42.473012    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp"
Oct 29 23:12:42.545979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:42.544050    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp" podStartSLOduration=2.544029641 podStartE2EDuration="2.544029641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:42.534831845 +0000 UTC m=+1765.877780633" watchObservedRunningTime="2026-10-29 23:12:42.544029641 +0000 UTC m=+1765.886978381"
Oct 29 23:12:42.854880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:42.852110    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d" path="/var/lib/kubelet/pods/9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d/volumes"
Oct 29 23:12:42.871824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:42.867205    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 29 23:12:42.871824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:42.867281    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj" podUID="8e3f78e3-9804-4b4f-8e9d-6e37b0351643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 29 23:12:42.890884 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:42.878466    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:42.970826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:42.953916    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Oct 29 23:12:43.309852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:43.297835    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:43.544058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:43.543261    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 29 23:12:43.561206 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:43.560921    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp" event={"ID":"b73ad056-0a89-4c23-84b7-d0eceab57c0b","Type":"ContainerStarted","Data":"a55be33c86914b092d49c10b843ab03c9e2ff7abe48cb092c1b69dbdd10eb2ae"}
Oct 29 23:12:43.641191 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:43.641142    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:43.676922 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:43.676880    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:43.742138 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:43.742097    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:43.900945 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:43.895083    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:43.974835 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:43.966742    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 29 23:12:43.974835 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:43.966820    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj" podUID="8e3f78e3-9804-4b4f-8e9d-6e37b0351643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 29 23:12:43.974835 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:43.968205    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:43.991939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:43.976725    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 23:12:44.217913 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:44.215922    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded"
Oct 29 23:12:44.347856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:44.314187    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:44.347856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:44.314619    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 23:12:44.421505 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:44.387786    5428 patch_prober.go:29] interesting pod/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:12:44.421505 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:44.387866    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4" podUID="d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:44.622972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:44.622922    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:44.639855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:44.639821    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 23:12:44.653870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:44.645413    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:44.653870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:44.645903    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 23:12:44.678857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:44.677291    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:44.678857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:44.678580    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 23:12:44.743004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:44.742941    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:44.758871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:44.752117    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 23:12:44.975928 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:44.973746    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 29 23:12:44.975928 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:44.973822    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj" podUID="8e3f78e3-9804-4b4f-8e9d-6e37b0351643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 29 23:12:44.985841 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:44.977287    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:44.985841 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:44.978912    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 29 23:12:45.051578 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:45.041353    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 23:12:45.233891 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:45.231543    5428 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:45.263825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:45.258267    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 23:12:45.317826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:45.317022    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:45.392195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:45.365602    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 23:12:45.559186 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:45.545011    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 23:12:45.602824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:45.601764    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 23:12:45.631830 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:45.624076    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:12:45.641829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:45.635511    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 23:12:45.741371 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:45.729531    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 23:12:45.741371 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:45.741149    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 23:12:45.883856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:45.882608    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 29 23:12:45.883856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:45.882665    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj" podUID="8e3f78e3-9804-4b4f-8e9d-6e37b0351643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 29 23:12:46.441288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:46.441254    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:12:46.862575 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:46.862482    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 29 23:12:46.862575 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:46.862551    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj" podUID="8e3f78e3-9804-4b4f-8e9d-6e37b0351643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 29 23:12:47.372841 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:12:47.372786    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:12:47.638355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:47.638260    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:48.717638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:48.717603    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:48.718161 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:48.717988    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:48.718746 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:48.718730    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:12:48.802900 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:48.802867    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:48.929941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:48.929884    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:49.190922 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.190886    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:49.258753 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.258717    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"]
Oct 29 23:12:49.259020 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.258970    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6" podUID="e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a" containerName="maas-api" containerID="cri-o://39600d9ecaeb2aa3f658f72407c5e0f2ae126ce2a36a8c207c1468c18241e861" gracePeriod=30
Oct 29 23:12:49.367760 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.367666    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:49.663049 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.663019    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"
Oct 29 23:12:49.717967 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.717928    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-openshift-service-ca\") pod \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\" (UID: \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\") "
Oct 29 23:12:49.718459 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.718018    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-kube-api-access-m55x2\" (UniqueName: \"kubernetes.io/projected/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-kube-api-access-m55x2\") pod \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\" (UID: \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\") "
Oct 29 23:12:49.718459 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.718091    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-metrics-certs\") pod \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\" (UID: \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\") "
Oct 29 23:12:49.718459 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.718327    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-maas-api-tls\") pod \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\" (UID: \"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a\") "
Oct 29 23:12:49.719786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.719737    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-openshift-service-ca" pod "e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a" (UID: "e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:12:49.723886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.723272    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-metrics-certs" pod "e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a" (UID: "e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:12:49.723886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.723517    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-maas-api-tls" pod "e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a" (UID: "e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:12:49.737152 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.737007    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-kube-api-access-m55x2" pod "e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a" (UID: "e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a"). InnerVolumeSpecName "kube-api-access-m55x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:12:49.822315 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.822274    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:12:49.822315 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.822308    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:12:49.822556 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.822326    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m55x2\" (UniqueName: \"kubernetes.io/projected/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-kube-api-access-m55x2\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:12:49.822556 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.822340    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:12:49.882914 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.882875    5428 generic.go:356] "Generic (PLEG): container finished" podID="e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a" containerID="39600d9ecaeb2aa3f658f72407c5e0f2ae126ce2a36a8c207c1468c18241e861" exitCode=0
Oct 29 23:12:49.883126 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.882946    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"
Oct 29 23:12:49.883126 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.882963    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6" event={"ID":"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a","Type":"ContainerDied","Data":"39600d9ecaeb2aa3f658f72407c5e0f2ae126ce2a36a8c207c1468c18241e861"}
Oct 29 23:12:49.883126 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.883063    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6" event={"ID":"e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a","Type":"ContainerDied","Data":"b699335fab9fffc0556caf53b6f4bb05aa6176d09529b48ce24792e01718e76f"}
Oct 29 23:12:49.883126 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.883098    5428 scope.go:122] "RemoveContainer" containerID="39600d9ecaeb2aa3f658f72407c5e0f2ae126ce2a36a8c207c1468c18241e861"
Oct 29 23:12:49.897024 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.897004    5428 scope.go:122] "RemoveContainer" containerID="39600d9ecaeb2aa3f658f72407c5e0f2ae126ce2a36a8c207c1468c18241e861"
Oct 29 23:12:49.897349 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:12:49.897291    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39600d9ecaeb2aa3f658f72407c5e0f2ae126ce2a36a8c207c1468c18241e861\": container with ID starting with 39600d9ecaeb2aa3f658f72407c5e0f2ae126ce2a36a8c207c1468c18241e861 not found: ID does not exist" containerID="39600d9ecaeb2aa3f658f72407c5e0f2ae126ce2a36a8c207c1468c18241e861"
Oct 29 23:12:49.897490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.897358    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39600d9ecaeb2aa3f658f72407c5e0f2ae126ce2a36a8c207c1468c18241e861"} err="failed to get container status \"39600d9ecaeb2aa3f658f72407c5e0f2ae126ce2a36a8c207c1468c18241e861\": rpc error: code = NotFound desc = could not find container \"39600d9ecaeb2aa3f658f72407c5e0f2ae126ce2a36a8c207c1468c18241e861\": container with ID starting with 39600d9ecaeb2aa3f658f72407c5e0f2ae126ce2a36a8c207c1468c18241e861 not found: ID does not exist"
Oct 29 23:12:49.940004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.939971    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"]
Oct 29 23:12:49.944302 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:49.944267    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-65ffcb-c6fcdc58c-rknm6"]
Oct 29 23:12:50.413833 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:50.413766    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:50.529865 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:50.529833    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:50.779254 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:50.779182    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a" path="/var/lib/kubelet/pods/e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a/volumes"
Oct 29 23:12:51.096751 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:51.096662    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:52.630887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:52.630853    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:54.567094 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.567056    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"]
Oct 29 23:12:54.567792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.567777    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d" containerName="maas-api"
Oct 29 23:12:54.567862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.567795    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d" containerName="maas-api"
Oct 29 23:12:54.567899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.567876    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a" containerName="maas-api"
Oct 29 23:12:54.567899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.567882    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a" containerName="maas-api"
Oct 29 23:12:54.568009 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.567998    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f431b2f-78fd-43ea-bcbf-3d6a4a894f1d" containerName="maas-api"
Oct 29 23:12:54.568049 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.568019    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a0ef9c-08ce-4f3b-bcbd-db4001c4e49a" containerName="maas-api"
Oct 29 23:12:54.572956 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.572936    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"
Oct 29 23:12:54.601526 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.601500    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-b4b5e3-serving-cert\""
Oct 29 23:12:54.607489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.607463    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"]
Oct 29 23:12:54.685068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.685036    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:54.701503 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.701476    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92936ba4-7ef1-451e-9322-7cf3583600f5-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq\" (UID: \"92936ba4-7ef1-451e-9322-7cf3583600f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"
Oct 29 23:12:54.701637 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.701511    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92936ba4-7ef1-451e-9322-7cf3583600f5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq\" (UID: \"92936ba4-7ef1-451e-9322-7cf3583600f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"
Oct 29 23:12:54.701637 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.701542    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92936ba4-7ef1-451e-9322-7cf3583600f5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq\" (UID: \"92936ba4-7ef1-451e-9322-7cf3583600f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"
Oct 29 23:12:54.701856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.701834    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf9kv\" (UniqueName: \"kubernetes.io/projected/92936ba4-7ef1-451e-9322-7cf3583600f5-kube-api-access-rf9kv\") pod \"maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq\" (UID: \"92936ba4-7ef1-451e-9322-7cf3583600f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"
Oct 29 23:12:54.803336 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.803298    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf9kv\" (UniqueName: \"kubernetes.io/projected/92936ba4-7ef1-451e-9322-7cf3583600f5-kube-api-access-rf9kv\") pod \"maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq\" (UID: \"92936ba4-7ef1-451e-9322-7cf3583600f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"
Oct 29 23:12:54.803510 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.803428    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92936ba4-7ef1-451e-9322-7cf3583600f5-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq\" (UID: \"92936ba4-7ef1-451e-9322-7cf3583600f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"
Oct 29 23:12:54.803510 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.803450    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92936ba4-7ef1-451e-9322-7cf3583600f5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq\" (UID: \"92936ba4-7ef1-451e-9322-7cf3583600f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"
Oct 29 23:12:54.803510 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.803474    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92936ba4-7ef1-451e-9322-7cf3583600f5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq\" (UID: \"92936ba4-7ef1-451e-9322-7cf3583600f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"
Oct 29 23:12:54.804231 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.804202    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92936ba4-7ef1-451e-9322-7cf3583600f5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq\" (UID: \"92936ba4-7ef1-451e-9322-7cf3583600f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"
Oct 29 23:12:54.805976 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.805952    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92936ba4-7ef1-451e-9322-7cf3583600f5-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq\" (UID: \"92936ba4-7ef1-451e-9322-7cf3583600f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"
Oct 29 23:12:54.806182 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.806158    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92936ba4-7ef1-451e-9322-7cf3583600f5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq\" (UID: \"92936ba4-7ef1-451e-9322-7cf3583600f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"
Oct 29 23:12:54.831889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.831782    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf9kv\" (UniqueName: \"kubernetes.io/projected/92936ba4-7ef1-451e-9322-7cf3583600f5-kube-api-access-rf9kv\") pod \"maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq\" (UID: \"92936ba4-7ef1-451e-9322-7cf3583600f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"
Oct 29 23:12:54.842650 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.842614    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:54.887269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.887232    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"
Oct 29 23:12:54.981575 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:54.981518    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp"
Oct 29 23:12:55.019730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:55.019692    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:55.224226 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:55.224201    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"]
Oct 29 23:12:55.226204 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:12:55.226154    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92936ba4_7ef1_451e_9322_7cf3583600f5.slice/crio-1ddec55a416fddf86eb07027a0d100236de637517b161364b51a6a277b5e8d66 WatchSource:0}: Error finding container 1ddec55a416fddf86eb07027a0d100236de637517b161364b51a6a277b5e8d66: Status 404 returned error can't find the container with id 1ddec55a416fddf86eb07027a0d100236de637517b161364b51a6a277b5e8d66
Oct 29 23:12:55.952239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:55.923488    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:55.984906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:55.984792    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq" event={"ID":"92936ba4-7ef1-451e-9322-7cf3583600f5","Type":"ContainerStarted","Data":"97fe32b57861dad5bb1999c67c5d0d347fd77896bf746c321f4998c3b2ae8f59"}
Oct 29 23:12:55.984906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:55.984897    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq" event={"ID":"92936ba4-7ef1-451e-9322-7cf3583600f5","Type":"ContainerStarted","Data":"1ddec55a416fddf86eb07027a0d100236de637517b161364b51a6a277b5e8d66"}
Oct 29 23:12:55.985349 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:55.984928    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"
Oct 29 23:12:56.033617 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:56.033568    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq" podStartSLOduration=1.727137033 podStartE2EDuration="2.03355377s" totalImagesPullingTime="306.416737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:56.030599168 +0000 UTC m=+1779.373547913" watchObservedRunningTime="2026-10-29 23:12:56.03355377 +0000 UTC m=+1779.376502514"
Oct 29 23:12:56.469636 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:56.469606    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:56.595864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:56.593862    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:57.083057 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:57.083026    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:57.587020 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:57.586985    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:57.822598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:57.822569    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:57.977600 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:57.977562    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"]
Oct 29 23:12:57.977875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:57.977842    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w" podUID="854e3883-53e9-4c18-ac91-bba8f9d4987e" containerName="istio-proxy" containerID="cri-o://f43b76168bc8e8a51e0f255edcc1333a243118c119746fda5c66222637254321" gracePeriod=30
Oct 29 23:12:58.014705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:58.014676    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:59.280688 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:12:59.280659    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:00.754822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:00.754762    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:00.873236 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:00.873206    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:01.163507 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:01.163477    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:01.997930 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:01.997899    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"
Oct 29 23:13:03.014035 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:13:03.014005    5428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854e3883_53e9_4c18_ac91_bba8f9d4987e.slice/crio-f43b76168bc8e8a51e0f255edcc1333a243118c119746fda5c66222637254321.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 23:13:03.014418 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:13:03.014074    5428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854e3883_53e9_4c18_ac91_bba8f9d4987e.slice/crio-f43b76168bc8e8a51e0f255edcc1333a243118c119746fda5c66222637254321.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 23:13:03.086657 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.086623    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:03.347759 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.347731    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:13:03.511438 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.511400    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-token\" (UniqueName: \"kubernetes.io/projected/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-token\") pod \"854e3883-53e9-4c18-ac91-bba8f9d4987e\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") "
Oct 29 23:13:03.511438 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.511443    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-envoy\") pod \"854e3883-53e9-4c18-ac91-bba8f9d4987e\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") "
Oct 29 23:13:03.511668 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.511477    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-workload-certs\") pod \"854e3883-53e9-4c18-ac91-bba8f9d4987e\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") "
Oct 29 23:13:03.511668 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.511507    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-workload-socket\") pod \"854e3883-53e9-4c18-ac91-bba8f9d4987e\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") "
Oct 29 23:13:03.511668 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.511552    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-podinfo\") pod \"854e3883-53e9-4c18-ac91-bba8f9d4987e\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") "
Oct 29 23:13:03.511668 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.511652    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-data\") pod \"854e3883-53e9-4c18-ac91-bba8f9d4987e\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") "
Oct 29 23:13:03.511905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.511702    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-credential-socket\") pod \"854e3883-53e9-4c18-ac91-bba8f9d4987e\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") "
Oct 29 23:13:03.511905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.511853    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-workload-certs" pod "854e3883-53e9-4c18-ac91-bba8f9d4987e" (UID: "854e3883-53e9-4c18-ac91-bba8f9d4987e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:13:03.511905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.511868    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-workload-socket" pod "854e3883-53e9-4c18-ac91-bba8f9d4987e" (UID: "854e3883-53e9-4c18-ac91-bba8f9d4987e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:13:03.511905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.511874    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/854e3883-53e9-4c18-ac91-bba8f9d4987e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854e3883-53e9-4c18-ac91-bba8f9d4987e-istiod-ca-cert\") pod \"854e3883-53e9-4c18-ac91-bba8f9d4987e\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") "
Oct 29 23:13:03.512125 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.511946    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854e3883-53e9-4c18-ac91-bba8f9d4987e-kube-api-access-rjhd7\" (UniqueName: \"kubernetes.io/projected/854e3883-53e9-4c18-ac91-bba8f9d4987e-kube-api-access-rjhd7\") pod \"854e3883-53e9-4c18-ac91-bba8f9d4987e\" (UID: \"854e3883-53e9-4c18-ac91-bba8f9d4987e\") "
Oct 29 23:13:03.512125 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.511987    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-data" pod "854e3883-53e9-4c18-ac91-bba8f9d4987e" (UID: "854e3883-53e9-4c18-ac91-bba8f9d4987e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:13:03.512125 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.512087    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-credential-socket" pod "854e3883-53e9-4c18-ac91-bba8f9d4987e" (UID: "854e3883-53e9-4c18-ac91-bba8f9d4987e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:13:03.512240 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.512155    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/854e3883-53e9-4c18-ac91-bba8f9d4987e-istiod-ca-cert" pod "854e3883-53e9-4c18-ac91-bba8f9d4987e" (UID: "854e3883-53e9-4c18-ac91-bba8f9d4987e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:13:03.512590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.512569    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:13:03.512704 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.512592    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:13:03.512704 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.512602    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:13:03.512704 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.512613    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:13:03.512704 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.512621    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854e3883-53e9-4c18-ac91-bba8f9d4987e-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:13:03.513795 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.513771    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-envoy" pod "854e3883-53e9-4c18-ac91-bba8f9d4987e" (UID: "854e3883-53e9-4c18-ac91-bba8f9d4987e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:13:03.514149 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.514120    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854e3883-53e9-4c18-ac91-bba8f9d4987e-kube-api-access-rjhd7" pod "854e3883-53e9-4c18-ac91-bba8f9d4987e" (UID: "854e3883-53e9-4c18-ac91-bba8f9d4987e"). InnerVolumeSpecName "kube-api-access-rjhd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:13:03.514238 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.514210    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-token" pod "854e3883-53e9-4c18-ac91-bba8f9d4987e" (UID: "854e3883-53e9-4c18-ac91-bba8f9d4987e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:13:03.514306 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.514283    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-podinfo" pod "854e3883-53e9-4c18-ac91-bba8f9d4987e" (UID: "854e3883-53e9-4c18-ac91-bba8f9d4987e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:13:03.613584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.613510    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:13:03.613584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.613536    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:13:03.613584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.613547    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854e3883-53e9-4c18-ac91-bba8f9d4987e-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:13:03.613584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:03.613557    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjhd7\" (UniqueName: \"kubernetes.io/projected/854e3883-53e9-4c18-ac91-bba8f9d4987e-kube-api-access-rjhd7\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:13:04.043620 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.043581    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:04.047863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.047832    5428 generic.go:356] "Generic (PLEG): container finished" podID="854e3883-53e9-4c18-ac91-bba8f9d4987e" containerID="f43b76168bc8e8a51e0f255edcc1333a243118c119746fda5c66222637254321" exitCode=0
Oct 29 23:13:04.048035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.047898    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"
Oct 29 23:13:04.048035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.047835    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w" event={"ID":"854e3883-53e9-4c18-ac91-bba8f9d4987e","Type":"ContainerDied","Data":"f43b76168bc8e8a51e0f255edcc1333a243118c119746fda5c66222637254321"}
Oct 29 23:13:04.048153 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.048034    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w" event={"ID":"854e3883-53e9-4c18-ac91-bba8f9d4987e","Type":"ContainerDied","Data":"eb7fd5512bd01a8e1ee8cd0962e31296dc047a893280eb9759b20c9ae37e5b2a"}
Oct 29 23:13:04.048366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.048231    5428 scope.go:122] "RemoveContainer" containerID="f43b76168bc8e8a51e0f255edcc1333a243118c119746fda5c66222637254321"
Oct 29 23:13:04.077378 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.077353    5428 scope.go:122] "RemoveContainer" containerID="fa3a7376c623bfade6df0807bb3d645174ccd8ed0a5e783be3e71798832e4608"
Oct 29 23:13:04.087164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.087143    5428 scope.go:122] "RemoveContainer" containerID="f43b76168bc8e8a51e0f255edcc1333a243118c119746fda5c66222637254321"
Oct 29 23:13:04.087458 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:13:04.087441    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f43b76168bc8e8a51e0f255edcc1333a243118c119746fda5c66222637254321\": container with ID starting with f43b76168bc8e8a51e0f255edcc1333a243118c119746fda5c66222637254321 not found: ID does not exist" containerID="f43b76168bc8e8a51e0f255edcc1333a243118c119746fda5c66222637254321"
Oct 29 23:13:04.087503 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.087466    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f43b76168bc8e8a51e0f255edcc1333a243118c119746fda5c66222637254321"} err="failed to get container status \"f43b76168bc8e8a51e0f255edcc1333a243118c119746fda5c66222637254321\": rpc error: code = NotFound desc = could not find container \"f43b76168bc8e8a51e0f255edcc1333a243118c119746fda5c66222637254321\": container with ID starting with f43b76168bc8e8a51e0f255edcc1333a243118c119746fda5c66222637254321 not found: ID does not exist"
Oct 29 23:13:04.087503 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.087484    5428 scope.go:122] "RemoveContainer" containerID="fa3a7376c623bfade6df0807bb3d645174ccd8ed0a5e783be3e71798832e4608"
Oct 29 23:13:04.087778 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:13:04.087742    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3a7376c623bfade6df0807bb3d645174ccd8ed0a5e783be3e71798832e4608\": container with ID starting with fa3a7376c623bfade6df0807bb3d645174ccd8ed0a5e783be3e71798832e4608 not found: ID does not exist" containerID="fa3a7376c623bfade6df0807bb3d645174ccd8ed0a5e783be3e71798832e4608"
Oct 29 23:13:04.087844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.087785    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3a7376c623bfade6df0807bb3d645174ccd8ed0a5e783be3e71798832e4608"} err="failed to get container status \"fa3a7376c623bfade6df0807bb3d645174ccd8ed0a5e783be3e71798832e4608\": rpc error: code = NotFound desc = could not find container \"fa3a7376c623bfade6df0807bb3d645174ccd8ed0a5e783be3e71798832e4608\": container with ID starting with fa3a7376c623bfade6df0807bb3d645174ccd8ed0a5e783be3e71798832e4608 not found: ID does not exist"
Oct 29 23:13:04.123772 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.123740    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"]
Oct 29 23:13:04.137434 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.137404    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-65ffcb-openshift-default-556d4476d5-f876w"]
Oct 29 23:13:04.153752 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.153725    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4"]
Oct 29 23:13:04.154436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.154420    5428 cpu_manager.go:414] "containerMap: removing container" podUID="854e3883-53e9-4c18-ac91-bba8f9d4987e" containerName="istio-proxy"
Oct 29 23:13:04.154486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.154437    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="854e3883-53e9-4c18-ac91-bba8f9d4987e" containerName="istio-proxy"
Oct 29 23:13:04.154486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.154465    5428 cpu_manager.go:414] "containerMap: removing container" podUID="854e3883-53e9-4c18-ac91-bba8f9d4987e" containerName="istio-proxy"
Oct 29 23:13:04.154486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.154470    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="854e3883-53e9-4c18-ac91-bba8f9d4987e" containerName="istio-proxy"
Oct 29 23:13:04.154581 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.154567    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="854e3883-53e9-4c18-ac91-bba8f9d4987e" containerName="istio-proxy"
Oct 29 23:13:04.154619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.154580    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="854e3883-53e9-4c18-ac91-bba8f9d4987e" containerName="istio-proxy"
Oct 29 23:13:04.159626 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.159603    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4"
Oct 29 23:13:04.175145 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.175121    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-7a3c1a\""
Oct 29 23:13:04.175293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.175198    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-7a3c1a-dockercfg-cpvms\""
Oct 29 23:13:04.178709 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.178686    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4"]
Oct 29 23:13:04.179173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.179151    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:04.216620 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.216586    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj"]
Oct 29 23:13:04.221789 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.221619    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj"
Oct 29 23:13:04.246148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.246108    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj"]
Oct 29 23:13:04.326682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.326570    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db877578-1ef2-4a99-b585-0ea0504a9613-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4\" (UID: \"db877578-1ef2-4a99-b585-0ea0504a9613\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4"
Oct 29 23:13:04.327107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.327079    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbsjz\" (UniqueName: \"kubernetes.io/projected/b42c20e6-b20d-4380-a772-b06a9b3d3592-kube-api-access-kbsjz\") pod \"payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj\" (UID: \"b42c20e6-b20d-4380-a772-b06a9b3d3592\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj"
Oct 29 23:13:04.327413 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.327396    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b42c20e6-b20d-4380-a772-b06a9b3d3592-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj\" (UID: \"b42c20e6-b20d-4380-a772-b06a9b3d3592\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj"
Oct 29 23:13:04.327728 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.327695    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blrmx\" (UniqueName: \"kubernetes.io/projected/db877578-1ef2-4a99-b585-0ea0504a9613-kube-api-access-blrmx\") pod \"payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4\" (UID: \"db877578-1ef2-4a99-b585-0ea0504a9613\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4"
Oct 29 23:13:04.428766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.428729    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbsjz\" (UniqueName: \"kubernetes.io/projected/b42c20e6-b20d-4380-a772-b06a9b3d3592-kube-api-access-kbsjz\") pod \"payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj\" (UID: \"b42c20e6-b20d-4380-a772-b06a9b3d3592\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj"
Oct 29 23:13:04.429003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.428847    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b42c20e6-b20d-4380-a772-b06a9b3d3592-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj\" (UID: \"b42c20e6-b20d-4380-a772-b06a9b3d3592\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj"
Oct 29 23:13:04.429003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.428938    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blrmx\" (UniqueName: \"kubernetes.io/projected/db877578-1ef2-4a99-b585-0ea0504a9613-kube-api-access-blrmx\") pod \"payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4\" (UID: \"db877578-1ef2-4a99-b585-0ea0504a9613\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4"
Oct 29 23:13:04.429095 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.429083    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db877578-1ef2-4a99-b585-0ea0504a9613-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4\" (UID: \"db877578-1ef2-4a99-b585-0ea0504a9613\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4"
Oct 29 23:13:04.429631 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.429612    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b42c20e6-b20d-4380-a772-b06a9b3d3592-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj\" (UID: \"b42c20e6-b20d-4380-a772-b06a9b3d3592\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj"
Oct 29 23:13:04.429721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.429703    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db877578-1ef2-4a99-b585-0ea0504a9613-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4\" (UID: \"db877578-1ef2-4a99-b585-0ea0504a9613\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4"
Oct 29 23:13:04.454118 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.454084    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbsjz\" (UniqueName: \"kubernetes.io/projected/b42c20e6-b20d-4380-a772-b06a9b3d3592-kube-api-access-kbsjz\") pod \"payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj\" (UID: \"b42c20e6-b20d-4380-a772-b06a9b3d3592\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj"
Oct 29 23:13:04.454118 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.454109    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blrmx\" (UniqueName: \"kubernetes.io/projected/db877578-1ef2-4a99-b585-0ea0504a9613-kube-api-access-blrmx\") pod \"payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4\" (UID: \"db877578-1ef2-4a99-b585-0ea0504a9613\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4"
Oct 29 23:13:04.474020 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.473996    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4"
Oct 29 23:13:04.548592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.548552    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj"
Oct 29 23:13:04.779585 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.779552    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="854e3883-53e9-4c18-ac91-bba8f9d4987e" path="/var/lib/kubelet/pods/854e3883-53e9-4c18-ac91-bba8f9d4987e/volumes"
Oct 29 23:13:04.824863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.824836    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4"]
Oct 29 23:13:04.825374 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:13:04.825349    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb877578_1ef2_4a99_b585_0ea0504a9613.slice/crio-4dd9e228e57ce41999be85a718a0a05f6a938c691a72164bf56ae4242b6c756b WatchSource:0}: Error finding container 4dd9e228e57ce41999be85a718a0a05f6a938c691a72164bf56ae4242b6c756b: Status 404 returned error can't find the container with id 4dd9e228e57ce41999be85a718a0a05f6a938c691a72164bf56ae4242b6c756b
Oct 29 23:13:04.930308 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:04.930283    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj"]
Oct 29 23:13:04.931181 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:13:04.931159    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42c20e6_b20d_4380_a772_b06a9b3d3592.slice/crio-a24dfed56b54feb337b7e73fb50235934e0574edea6340b0e03901c52a8c80ac WatchSource:0}: Error finding container a24dfed56b54feb337b7e73fb50235934e0574edea6340b0e03901c52a8c80ac: Status 404 returned error can't find the container with id a24dfed56b54feb337b7e73fb50235934e0574edea6340b0e03901c52a8c80ac
Oct 29 23:13:05.058595 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:05.058558    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4" event={"ID":"db877578-1ef2-4a99-b585-0ea0504a9613","Type":"ContainerStarted","Data":"647140312bb15b4fc4adcea002553e23ceedcd070c9b6ef1a49c4ddf3ce7388b"}
Oct 29 23:13:05.059048 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:05.058628    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4"
Oct 29 23:13:05.059048 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:05.058652    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4" event={"ID":"db877578-1ef2-4a99-b585-0ea0504a9613","Type":"ContainerStarted","Data":"4dd9e228e57ce41999be85a718a0a05f6a938c691a72164bf56ae4242b6c756b"}
Oct 29 23:13:05.060437 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:05.060405    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj" event={"ID":"b42c20e6-b20d-4380-a772-b06a9b3d3592","Type":"ContainerStarted","Data":"6456f2646a10d5cef465fa484b53076e998ccd2c9523779f3df970dc8574fa1b"}
Oct 29 23:13:05.060574 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:05.060458    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj" event={"ID":"b42c20e6-b20d-4380-a772-b06a9b3d3592","Type":"ContainerStarted","Data":"a24dfed56b54feb337b7e73fb50235934e0574edea6340b0e03901c52a8c80ac"}
Oct 29 23:13:05.060947 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:05.060924    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj"
Oct 29 23:13:05.100153 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:05.100059    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4" podStartSLOduration=1.100044108 podStartE2EDuration="1.100044108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:05.098177299 +0000 UTC m=+1788.441126045" watchObservedRunningTime="2026-10-29 23:13:05.100044108 +0000 UTC m=+1788.442992854"
Oct 29 23:13:05.139858 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:05.139784    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj" podStartSLOduration=1.139767212 podStartE2EDuration="1.139767212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:05.138593454 +0000 UTC m=+1788.481542200" watchObservedRunningTime="2026-10-29 23:13:05.139767212 +0000 UTC m=+1788.482715958"
Oct 29 23:13:05.709332 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:05.709300    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:07.662599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:07.662562    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:09.322470 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:09.322411    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:09.442372 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:09.442340    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:09.572635 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:09.572543    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:10.771732 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:10.771678    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:11.195476 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:11.195447    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:11.321107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:11.321077    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:12.116354 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:12.116317    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:12.237093 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:12.237040    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:13.542314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:13.542286    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:15.021324 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:15.021294    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:15.476305 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:15.476275    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:15.613771 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:15.613720    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:15.920758 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:15.920719    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:16.093356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:16.088334    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4"
Oct 29 23:13:16.334645 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:16.334571    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:16.466029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:16.465995    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:16.600235 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:16.600149    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:16.737444 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:16.737409    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:17.080454 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:17.080424    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj"
Oct 29 23:13:17.158530 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:17.158496    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:17.331511 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:17.331411    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:17.521612 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:13:17.521576    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:13:18.285388 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.285348    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:18.400070 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.400031    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:18.447513 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.447477    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx"]
Oct 29 23:13:18.453346 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.453329    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx"
Oct 29 23:13:18.470981 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.470954    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-4ad17e\""
Oct 29 23:13:18.483166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.483144    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-4ad17e-dockercfg-lc2b7\""
Oct 29 23:13:18.487654 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.487632    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx"]
Oct 29 23:13:18.603761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.603088    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k"]
Oct 29 23:13:18.606889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.606855    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8035b7-d41e-43bf-842e-9f9f9e54eae9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx\" (UID: \"4b8035b7-d41e-43bf-842e-9f9f9e54eae9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx"
Oct 29 23:13:18.607315 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.607294    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvwtx\" (UniqueName: \"kubernetes.io/projected/4b8035b7-d41e-43bf-842e-9f9f9e54eae9-kube-api-access-xvwtx\") pod \"payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx\" (UID: \"4b8035b7-d41e-43bf-842e-9f9f9e54eae9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx"
Oct 29 23:13:18.618534 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.618503    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k"]
Oct 29 23:13:18.618683 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.618599    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k"
Oct 29 23:13:18.709527 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.709475    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp76h\" (UniqueName: \"kubernetes.io/projected/3d79a8de-da14-43a3-b4f1-cdb7fd66d235-kube-api-access-pp76h\") pod \"payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k\" (UID: \"3d79a8de-da14-43a3-b4f1-cdb7fd66d235\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k"
Oct 29 23:13:18.709742 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.709696    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d79a8de-da14-43a3-b4f1-cdb7fd66d235-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k\" (UID: \"3d79a8de-da14-43a3-b4f1-cdb7fd66d235\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k"
Oct 29 23:13:18.710295 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.709768    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8035b7-d41e-43bf-842e-9f9f9e54eae9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx\" (UID: \"4b8035b7-d41e-43bf-842e-9f9f9e54eae9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx"
Oct 29 23:13:18.710295 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.709940    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvwtx\" (UniqueName: \"kubernetes.io/projected/4b8035b7-d41e-43bf-842e-9f9f9e54eae9-kube-api-access-xvwtx\") pod \"payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx\" (UID: \"4b8035b7-d41e-43bf-842e-9f9f9e54eae9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx"
Oct 29 23:13:18.710794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.710767    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8035b7-d41e-43bf-842e-9f9f9e54eae9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx\" (UID: \"4b8035b7-d41e-43bf-842e-9f9f9e54eae9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx"
Oct 29 23:13:18.733469 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.733428    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvwtx\" (UniqueName: \"kubernetes.io/projected/4b8035b7-d41e-43bf-842e-9f9f9e54eae9-kube-api-access-xvwtx\") pod \"payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx\" (UID: \"4b8035b7-d41e-43bf-842e-9f9f9e54eae9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx"
Oct 29 23:13:18.767851 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.767795    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx"
Oct 29 23:13:18.811877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.811840    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp76h\" (UniqueName: \"kubernetes.io/projected/3d79a8de-da14-43a3-b4f1-cdb7fd66d235-kube-api-access-pp76h\") pod \"payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k\" (UID: \"3d79a8de-da14-43a3-b4f1-cdb7fd66d235\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k"
Oct 29 23:13:18.812058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.811956    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d79a8de-da14-43a3-b4f1-cdb7fd66d235-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k\" (UID: \"3d79a8de-da14-43a3-b4f1-cdb7fd66d235\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k"
Oct 29 23:13:18.812672 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.812647    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d79a8de-da14-43a3-b4f1-cdb7fd66d235-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k\" (UID: \"3d79a8de-da14-43a3-b4f1-cdb7fd66d235\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k"
Oct 29 23:13:18.843506 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.842401    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp76h\" (UniqueName: \"kubernetes.io/projected/3d79a8de-da14-43a3-b4f1-cdb7fd66d235-kube-api-access-pp76h\") pod \"payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k\" (UID: \"3d79a8de-da14-43a3-b4f1-cdb7fd66d235\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k"
Oct 29 23:13:18.957880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:18.957387    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k"
Oct 29 23:13:19.105254 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:19.103544    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx"]
Oct 29 23:13:19.111310 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:13:19.111275    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8035b7_d41e_43bf_842e_9f9f9e54eae9.slice/crio-c95befa41d8bfa82f2f603a963449d767d48854d9d5eaa835ede6f08efb00bc6 WatchSource:0}: Error finding container c95befa41d8bfa82f2f603a963449d767d48854d9d5eaa835ede6f08efb00bc6: Status 404 returned error can't find the container with id c95befa41d8bfa82f2f603a963449d767d48854d9d5eaa835ede6f08efb00bc6
Oct 29 23:13:19.187003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:19.186972    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx" event={"ID":"4b8035b7-d41e-43bf-842e-9f9f9e54eae9","Type":"ContainerStarted","Data":"c95befa41d8bfa82f2f603a963449d767d48854d9d5eaa835ede6f08efb00bc6"}
Oct 29 23:13:19.294179 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:19.294148    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k"]
Oct 29 23:13:19.295248 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:13:19.295222    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d79a8de_da14_43a3_b4f1_cdb7fd66d235.slice/crio-8d3429acc28b9d864bc0fc8393258390550d72fb02a4113c4e645c2edbb850b4 WatchSource:0}: Error finding container 8d3429acc28b9d864bc0fc8393258390550d72fb02a4113c4e645c2edbb850b4: Status 404 returned error can't find the container with id 8d3429acc28b9d864bc0fc8393258390550d72fb02a4113c4e645c2edbb850b4
Oct 29 23:13:20.200292 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:20.200249    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k" event={"ID":"3d79a8de-da14-43a3-b4f1-cdb7fd66d235","Type":"ContainerStarted","Data":"0fa64e86b0825984e8f80f7380e676982bac1b1884556e341cc89797ae0f7c26"}
Oct 29 23:13:20.200459 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:20.200307    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k" event={"ID":"3d79a8de-da14-43a3-b4f1-cdb7fd66d235","Type":"ContainerStarted","Data":"8d3429acc28b9d864bc0fc8393258390550d72fb02a4113c4e645c2edbb850b4"}
Oct 29 23:13:20.201261 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:20.201234    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k"
Oct 29 23:13:20.203010 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:20.202983    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx" event={"ID":"4b8035b7-d41e-43bf-842e-9f9f9e54eae9","Type":"ContainerStarted","Data":"d2da6df1945e029e8e3f20c6a337426d63bfdffa9d818025b7b0a0ae8239bf38"}
Oct 29 23:13:20.203094 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:20.203040    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx"
Oct 29 23:13:20.244775 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:20.244720    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k" podStartSLOduration=2.244706561 podStartE2EDuration="2.244706561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:20.240431719 +0000 UTC m=+1803.583380465" watchObservedRunningTime="2026-10-29 23:13:20.244706561 +0000 UTC m=+1803.587655305"
Oct 29 23:13:20.282760 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:20.282693    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx" podStartSLOduration=2.282675394 podStartE2EDuration="2.282675394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:20.280162907 +0000 UTC m=+1803.623111676" watchObservedRunningTime="2026-10-29 23:13:20.282675394 +0000 UTC m=+1803.625624141"
Oct 29 23:13:20.374665 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:20.374634    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:20.531723 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:20.531622    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:20.669032 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:20.668996    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:21.208824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:21.208180    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:21.356505 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:21.356473    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:22.403828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:22.386573    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:22.551546 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:22.551477    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:22.555925 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:22.555899    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5"]
Oct 29 23:13:22.660874 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:22.613261    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5"]
Oct 29 23:13:22.660874 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:22.613568    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5"
Oct 29 23:13:22.735825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:22.733307    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c83b50-dockercfg-jkhqf\""
Oct 29 23:13:22.735825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:22.733436    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c83b50\""
Oct 29 23:13:22.803826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:22.803179    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnbdd\" (UniqueName: \"kubernetes.io/projected/baf2a48c-3aa1-47fb-aa76-618f73d57163-kube-api-access-xnbdd\") pod \"payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5\" (UID: \"baf2a48c-3aa1-47fb-aa76-618f73d57163\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5"
Oct 29 23:13:22.803826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:22.803271    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf2a48c-3aa1-47fb-aa76-618f73d57163-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5\" (UID: \"baf2a48c-3aa1-47fb-aa76-618f73d57163\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5"
Oct 29 23:13:22.958886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:22.955041    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnbdd\" (UniqueName: \"kubernetes.io/projected/baf2a48c-3aa1-47fb-aa76-618f73d57163-kube-api-access-xnbdd\") pod \"payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5\" (UID: \"baf2a48c-3aa1-47fb-aa76-618f73d57163\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5"
Oct 29 23:13:22.958886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:22.955196    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf2a48c-3aa1-47fb-aa76-618f73d57163-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5\" (UID: \"baf2a48c-3aa1-47fb-aa76-618f73d57163\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5"
Oct 29 23:13:22.958886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:22.957639    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf2a48c-3aa1-47fb-aa76-618f73d57163-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5\" (UID: \"baf2a48c-3aa1-47fb-aa76-618f73d57163\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5"
Oct 29 23:13:23.019906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:23.019794    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnbdd\" (UniqueName: \"kubernetes.io/projected/baf2a48c-3aa1-47fb-aa76-618f73d57163-kube-api-access-xnbdd\") pod \"payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5\" (UID: \"baf2a48c-3aa1-47fb-aa76-618f73d57163\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5"
Oct 29 23:13:23.098826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:23.097162    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5"
Oct 29 23:13:23.730821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:23.730770    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5"]
Oct 29 23:13:23.731273 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:13:23.731096    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf2a48c_3aa1_47fb_aa76_618f73d57163.slice/crio-0f777f9aea3b8ddea7035a7eda87f7cd3fe055e79ef214456403f76b2bb02078 WatchSource:0}: Error finding container 0f777f9aea3b8ddea7035a7eda87f7cd3fe055e79ef214456403f76b2bb02078: Status 404 returned error can't find the container with id 0f777f9aea3b8ddea7035a7eda87f7cd3fe055e79ef214456403f76b2bb02078
Oct 29 23:13:24.364946 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.364905    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5" event={"ID":"baf2a48c-3aa1-47fb-aa76-618f73d57163","Type":"ContainerStarted","Data":"c30d61f68348f49e1355a58ff28970aa6e0ccc1ec1b003cd3282f725e3c65f35"}
Oct 29 23:13:24.365149 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.364975    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5"
Oct 29 23:13:24.365149 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.364998    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5" event={"ID":"baf2a48c-3aa1-47fb-aa76-618f73d57163","Type":"ContainerStarted","Data":"0f777f9aea3b8ddea7035a7eda87f7cd3fe055e79ef214456403f76b2bb02078"}
Oct 29 23:13:24.382583 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.382543    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:24.412033 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.411969    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5" podStartSLOduration=2.411954457 podStartE2EDuration="2.411954457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:24.406945432 +0000 UTC m=+1807.749894182" watchObservedRunningTime="2026-10-29 23:13:24.411954457 +0000 UTC m=+1807.754903204"
Oct 29 23:13:24.517969 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.517918    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:24.518903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.518877    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m"]
Oct 29 23:13:24.523392 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.523375    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m"
Oct 29 23:13:24.542486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.540071    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-c47a72\""
Oct 29 23:13:24.542486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.540329    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-c47a72-dockercfg-mjtz8\""
Oct 29 23:13:24.564972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.564932    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m"]
Oct 29 23:13:24.617881 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.616874    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzz57\" (UniqueName: \"kubernetes.io/projected/1b715bb1-d85c-4b85-87d0-8a67112e3d4f-kube-api-access-pzz57\") pod \"payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m\" (UID: \"1b715bb1-d85c-4b85-87d0-8a67112e3d4f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m"
Oct 29 23:13:24.617881 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.616962    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b715bb1-d85c-4b85-87d0-8a67112e3d4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m\" (UID: \"1b715bb1-d85c-4b85-87d0-8a67112e3d4f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m"
Oct 29 23:13:24.643825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.641951    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf"]
Oct 29 23:13:24.661915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.649666    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf"
Oct 29 23:13:24.661915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.658842    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf"]
Oct 29 23:13:24.747427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.745234    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf\" (UID: \"79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd\") " pod="openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf"
Oct 29 23:13:24.747427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.745319    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn6n9\" (UniqueName: \"kubernetes.io/projected/79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd-kube-api-access-zn6n9\") pod \"payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf\" (UID: \"79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd\") " pod="openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf"
Oct 29 23:13:24.747427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.745416    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzz57\" (UniqueName: \"kubernetes.io/projected/1b715bb1-d85c-4b85-87d0-8a67112e3d4f-kube-api-access-pzz57\") pod \"payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m\" (UID: \"1b715bb1-d85c-4b85-87d0-8a67112e3d4f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m"
Oct 29 23:13:24.747427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.745469    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b715bb1-d85c-4b85-87d0-8a67112e3d4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m\" (UID: \"1b715bb1-d85c-4b85-87d0-8a67112e3d4f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m"
Oct 29 23:13:24.747427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.746470    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b715bb1-d85c-4b85-87d0-8a67112e3d4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m\" (UID: \"1b715bb1-d85c-4b85-87d0-8a67112e3d4f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m"
Oct 29 23:13:24.773857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.771046    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzz57\" (UniqueName: \"kubernetes.io/projected/1b715bb1-d85c-4b85-87d0-8a67112e3d4f-kube-api-access-pzz57\") pod \"payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m\" (UID: \"1b715bb1-d85c-4b85-87d0-8a67112e3d4f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m"
Oct 29 23:13:24.839764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.839726    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m"
Oct 29 23:13:24.847298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.847270    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf\" (UID: \"79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd\") " pod="openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf"
Oct 29 23:13:24.847453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.847337    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6n9\" (UniqueName: \"kubernetes.io/projected/79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd-kube-api-access-zn6n9\") pod \"payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf\" (UID: \"79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd\") " pod="openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf"
Oct 29 23:13:24.847929 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.847909    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf\" (UID: \"79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd\") " pod="openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf"
Oct 29 23:13:24.871164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:24.871080    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn6n9\" (UniqueName: \"kubernetes.io/projected/79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd-kube-api-access-zn6n9\") pod \"payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf\" (UID: \"79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd\") " pod="openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf"
Oct 29 23:13:25.013739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:25.013708    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf"
Oct 29 23:13:25.158258 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:25.158225    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m"]
Oct 29 23:13:25.159033 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:13:25.159001    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b715bb1_d85c_4b85_87d0_8a67112e3d4f.slice/crio-a7a01578c5917cf2e9995e0c06692f2c3f200661186d944ee2936f411d865738 WatchSource:0}: Error finding container a7a01578c5917cf2e9995e0c06692f2c3f200661186d944ee2936f411d865738: Status 404 returned error can't find the container with id a7a01578c5917cf2e9995e0c06692f2c3f200661186d944ee2936f411d865738
Oct 29 23:13:25.330353 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:13:25.330304    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d1fa8d_832d_4a58_b7d6_b4cbd7e06cfd.slice/crio-0bf8088d941e2f5bf9c47724070bc926f25e107cddf7d47b4907df0a0a9abbfd WatchSource:0}: Error finding container 0bf8088d941e2f5bf9c47724070bc926f25e107cddf7d47b4907df0a0a9abbfd: Status 404 returned error can't find the container with id 0bf8088d941e2f5bf9c47724070bc926f25e107cddf7d47b4907df0a0a9abbfd
Oct 29 23:13:25.331121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:25.331093    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf"]
Oct 29 23:13:25.379001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:25.378964    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf" event={"ID":"79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd","Type":"ContainerStarted","Data":"0bf8088d941e2f5bf9c47724070bc926f25e107cddf7d47b4907df0a0a9abbfd"}
Oct 29 23:13:25.387099 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:25.386718    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m" event={"ID":"1b715bb1-d85c-4b85-87d0-8a67112e3d4f","Type":"ContainerStarted","Data":"a0dc2fef0f3e39b7a06309f5141f505ed4e3532c848a1e0ed482f84ed3430272"}
Oct 29 23:13:25.387099 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:25.386846    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m"
Oct 29 23:13:25.387099 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:25.386876    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m" event={"ID":"1b715bb1-d85c-4b85-87d0-8a67112e3d4f","Type":"ContainerStarted","Data":"a7a01578c5917cf2e9995e0c06692f2c3f200661186d944ee2936f411d865738"}
Oct 29 23:13:25.431556 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:25.431506    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m" podStartSLOduration=1.431489517 podStartE2EDuration="1.431489517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:25.428372029 +0000 UTC m=+1808.771320776" watchObservedRunningTime="2026-10-29 23:13:25.431489517 +0000 UTC m=+1808.774438266"
Oct 29 23:13:26.134723 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:26.134689    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:26.395055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:26.394956    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf" event={"ID":"79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd","Type":"ContainerStarted","Data":"a8bd3479d027cdf0e3bfb4aa6266e37e6d1cf480c143acc5b71dc6c61dc769c4"}
Oct 29 23:13:26.395193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:26.395051    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf"
Oct 29 23:13:26.436686 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:26.436635    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf" podStartSLOduration=2.436621809 podStartE2EDuration="2.436621809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:26.434886522 +0000 UTC m=+1809.777835267" watchObservedRunningTime="2026-10-29 23:13:26.436621809 +0000 UTC m=+1809.779570554"
Oct 29 23:13:27.940377 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:27.940301    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:28.166562 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:28.166528    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:28.545447 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:28.545413    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:28.685511 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:28.685473    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:29.603978 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:29.603915    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:30.845502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:30.845469    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:31.240555 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:31.240527    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx"
Oct 29 23:13:32.313103 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:32.313075    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k"
Oct 29 23:13:33.091520 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:33.091484    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:33.222456 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:33.222417    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:33.359419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:33.359324    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:33.481473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:33.481427    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:35.302924 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.302885    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"]
Oct 29 23:13:35.308733 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.308713    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"
Oct 29 23:13:35.325638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.325610    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-7a3c1a-serving-cert\""
Oct 29 23:13:35.333735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.333674    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"]
Oct 29 23:13:35.391139 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.391114    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5"
Oct 29 23:13:35.398545 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.398518    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:35.402509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.402479    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv\" (UID: \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"
Oct 29 23:13:35.402617 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.402593    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv\" (UID: \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"
Oct 29 23:13:35.402671 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.402629    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b47z6\" (UniqueName: \"kubernetes.io/projected/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-kube-api-access-b47z6\") pod \"maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv\" (UID: \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"
Oct 29 23:13:35.402730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.402717    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-metrics-certs\") pod \"maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv\" (UID: \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"
Oct 29 23:13:35.504681 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.504646    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv\" (UID: \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"
Oct 29 23:13:35.504900 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.504713    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv\" (UID: \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"
Oct 29 23:13:35.504900 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.504737    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b47z6\" (UniqueName: \"kubernetes.io/projected/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-kube-api-access-b47z6\") pod \"maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv\" (UID: \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"
Oct 29 23:13:35.504900 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.504783    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-metrics-certs\") pod \"maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv\" (UID: \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"
Oct 29 23:13:35.505366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.505340    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv\" (UID: \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"
Oct 29 23:13:35.507367 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.507339    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv\" (UID: \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"
Oct 29 23:13:35.507471 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.507373    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-metrics-certs\") pod \"maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv\" (UID: \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"
Oct 29 23:13:35.513872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.513846    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:35.528451 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.528426    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b47z6\" (UniqueName: \"kubernetes.io/projected/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-kube-api-access-b47z6\") pod \"maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv\" (UID: \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"
Oct 29 23:13:35.627257 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.627225    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"
Oct 29 23:13:35.630452 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.630419    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:35.965852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:35.965827    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"]
Oct 29 23:13:35.966797 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:13:35.966775    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddadecf77_4bd8_4ca8_bd71_3b2f36b62953.slice/crio-6ea967f060f6381a9ef7b7fa956cf6bba16e5df05844f4baa777ccbe707ef238 WatchSource:0}: Error finding container 6ea967f060f6381a9ef7b7fa956cf6bba16e5df05844f4baa777ccbe707ef238: Status 404 returned error can't find the container with id 6ea967f060f6381a9ef7b7fa956cf6bba16e5df05844f4baa777ccbe707ef238
Oct 29 23:13:36.130027 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.129997    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"]
Oct 29 23:13:36.136671 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.136645    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.163462 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.163424    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"]
Oct 29 23:13:36.192071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.192039    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-b4b5e3-openshift-default-dockercfg-td4sl\""
Oct 29 23:13:36.217075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.216988    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-credential-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.217250 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.217071    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-data\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.217250 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.217130    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a967af1-6e08-4983-9493-0292d3ea8aaa-istiod-ca-cert\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.217250 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.217213    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-envoy\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.217440 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.217346    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-workload-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.217501 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.217480    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt2j7\" (UniqueName: \"kubernetes.io/projected/1a967af1-6e08-4983-9493-0292d3ea8aaa-kube-api-access-tt2j7\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.217560 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.217524    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-token\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.217618 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.217572    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-podinfo\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.217618 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.217605    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-workload-certs\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.319309 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.319271    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-workload-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.319890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.319402    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt2j7\" (UniqueName: \"kubernetes.io/projected/1a967af1-6e08-4983-9493-0292d3ea8aaa-kube-api-access-tt2j7\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.319890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.319443    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-token\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.319890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.319481    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-podinfo\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.319890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.319509    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-workload-certs\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.319890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.319585    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-credential-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.319890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.319657    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-data\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.319890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.319726    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a967af1-6e08-4983-9493-0292d3ea8aaa-istiod-ca-cert\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.319890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.319774    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-workload-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.319890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.319844    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-envoy\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.320433 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.320093    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-workload-certs\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.320433 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.320213    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-credential-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.320793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.320740    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a967af1-6e08-4983-9493-0292d3ea8aaa-istiod-ca-cert\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.320984 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.320830    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-data\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.323095 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.323064    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-envoy\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.323982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.323958    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-podinfo\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.343991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.343956    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-token\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.344251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.344206    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt2j7\" (UniqueName: \"kubernetes.io/projected/1a967af1-6e08-4983-9493-0292d3ea8aaa-kube-api-access-tt2j7\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.398476 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.398451    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m"
Oct 29 23:13:36.464340 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.464314    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:36.489580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.489495    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv" event={"ID":"dadecf77-4bd8-4ca8-bd71-3b2f36b62953","Type":"ContainerStarted","Data":"506fcd29817cf986e434dcd91dc11f434476b62936da5f71ce2c5f8c03f312dc"}
Oct 29 23:13:36.489580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.489567    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv" event={"ID":"dadecf77-4bd8-4ca8-bd71-3b2f36b62953","Type":"ContainerStarted","Data":"6ea967f060f6381a9ef7b7fa956cf6bba16e5df05844f4baa777ccbe707ef238"}
Oct 29 23:13:36.489849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.489598    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"
Oct 29 23:13:36.537496 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.537430    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv" podStartSLOduration=1.108051401 podStartE2EDuration="1.537409673s" totalImagesPullingTime="429.358272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:36.533940582 +0000 UTC m=+1819.876889304" watchObservedRunningTime="2026-10-29 23:13:36.537409673 +0000 UTC m=+1819.880358419"
Oct 29 23:13:36.800712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.800679    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"]
Oct 29 23:13:36.801920 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:13:36.801880    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a967af1_6e08_4983_9493_0292d3ea8aaa.slice/crio-e64d6efb4bd9eaa0833aad9a1515c0731e62148448553ab85454116f211c2370 WatchSource:0}: Error finding container e64d6efb4bd9eaa0833aad9a1515c0731e62148448553ab85454116f211c2370: Status 404 returned error can't find the container with id e64d6efb4bd9eaa0833aad9a1515c0731e62148448553ab85454116f211c2370
Oct 29 23:13:36.805245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.805212    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:13:36.805344 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.805278    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:13:36.805344 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:36.805304    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:13:37.408023 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:37.407997    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf"
Oct 29 23:13:37.501088 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:37.501038    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr" event={"ID":"1a967af1-6e08-4983-9493-0292d3ea8aaa","Type":"ContainerStarted","Data":"db51ae4c11874fe0170c333811bfaadd220dc85b374e30b35b3a4522d9563fe5"}
Oct 29 23:13:37.501278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:37.501111    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr" event={"ID":"1a967af1-6e08-4983-9493-0292d3ea8aaa","Type":"ContainerStarted","Data":"e64d6efb4bd9eaa0833aad9a1515c0731e62148448553ab85454116f211c2370"}
Oct 29 23:13:37.546948 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:37.546894    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr" podStartSLOduration=1.5468795910000002 podStartE2EDuration="1.546879591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:37.545343292 +0000 UTC m=+1820.888292038" watchObservedRunningTime="2026-10-29 23:13:37.546879591 +0000 UTC m=+1820.889828335"
Oct 29 23:13:37.603859 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:37.603779    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:38.468152 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:38.468118    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:38.472936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:38.472904    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 29 23:13:38.473190 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:38.473163    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr" podUID="1a967af1-6e08-4983-9493-0292d3ea8aaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 29 23:13:39.474264 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:39.474217    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 29 23:13:39.474870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:39.474836    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr" podUID="1a967af1-6e08-4983-9493-0292d3ea8aaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 29 23:13:39.596152 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:39.596112    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:40.484055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:40.483917    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 29 23:13:40.484055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:40.483985    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr" podUID="1a967af1-6e08-4983-9493-0292d3ea8aaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 29 23:13:40.929726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:40.929693    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:41.383853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:41.380750    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:41.482823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:41.479149    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 29 23:13:41.482823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:41.479207    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr" podUID="1a967af1-6e08-4983-9493-0292d3ea8aaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 29 23:13:41.854210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:41.854136    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:41.993139 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:41.993103    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:42.481824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:42.467158    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 29 23:13:42.481824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:42.467219    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr" podUID="1a967af1-6e08-4983-9493-0292d3ea8aaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 29 23:13:42.506534 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:42.506508    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"
Oct 29 23:13:42.895222 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:42.893772    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"]
Oct 29 23:13:42.900009 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:42.899969    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:42.901305 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:42.900406    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"]
Oct 29 23:13:42.952018 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:42.951985    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:42.952183 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:42.952121    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zvmg\" (UniqueName: \"kubernetes.io/projected/2e3ed106-1a9a-43e6-998f-52b37686da76-kube-api-access-8zvmg\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:42.952229 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:42.952183    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:42.952270 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:42.952245    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:42.952313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:42.952267    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:42.952387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:42.952374    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:42.952423 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:42.952398    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:42.952423 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:42.952413    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3ed106-1a9a-43e6-998f-52b37686da76-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:42.952525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:42.952511    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.053766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.053730    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvmg\" (UniqueName: \"kubernetes.io/projected/2e3ed106-1a9a-43e6-998f-52b37686da76-kube-api-access-8zvmg\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.053995 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.053784    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.053995 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.053869    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.053995 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.053891    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.054231 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.054014    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.054231 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.054046    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.054231 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.054070    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3ed106-1a9a-43e6-998f-52b37686da76-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.054231 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.054161    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.054447 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.054283    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-workload-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.054447 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.054304    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-workload-certs\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.054567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.054487    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-credential-socket\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.054623 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.054600    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.054955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.054931    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3ed106-1a9a-43e6-998f-52b37686da76-istiod-ca-cert\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.055042 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.054932    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-data\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.056940 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.056917    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-envoy\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.057246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.057226    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-podinfo\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.087736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.087697    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvmg\" (UniqueName: \"kubernetes.io/projected/2e3ed106-1a9a-43e6-998f-52b37686da76-kube-api-access-8zvmg\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.088783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.088764    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-token\") pod \"e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") " pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.218638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.218549    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:43.238324 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.238205    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:43.469517 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.469441    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:43.469881 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.469846    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:43.470575 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.470554    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:13:43.590763 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.590713    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:13:43.591043 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.591012    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:13:43.591172 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.591095    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:13:43.617341 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:43.617308    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"]
Oct 29 23:13:44.496772 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:44.496737    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:44.591520 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:44.591481    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf" event={"ID":"2e3ed106-1a9a-43e6-998f-52b37686da76","Type":"ContainerStarted","Data":"b0f94ed9cf5def08600d9b4815074b7c0d3e700d583ecf77c00592361c34b099"}
Oct 29 23:13:44.591724 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:44.591543    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf" event={"ID":"2e3ed106-1a9a-43e6-998f-52b37686da76","Type":"ContainerStarted","Data":"4a7d9b840a24e753d51de461d9e93dd3ca371664783cde7adbdee8cfc89acf1b"}
Oct 29 23:13:44.639097 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:44.639043    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf" podStartSLOduration=2.639030311 podStartE2EDuration="2.639030311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:44.638209721 +0000 UTC m=+1827.981158469" watchObservedRunningTime="2026-10-29 23:13:44.639030311 +0000 UTC m=+1827.981979056"
Oct 29 23:13:45.219149 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:45.219113    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:45.220778 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:45.220748    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 29 23:13:45.220929 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:45.220792    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf" podUID="2e3ed106-1a9a-43e6-998f-52b37686da76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 29 23:13:45.400984 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:45.400957    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:45.784735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:45.784703    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:45.913743 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:45.913711    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:46.180826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:46.180625    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:46.219538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:46.219504    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 29 23:13:46.219731 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:46.219573    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf" podUID="2e3ed106-1a9a-43e6-998f-52b37686da76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 29 23:13:47.220000 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:47.219958    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 29 23:13:47.220451 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:47.220018    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf" podUID="2e3ed106-1a9a-43e6-998f-52b37686da76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 29 23:13:47.391237 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:13:47.391206    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:13:47.693082 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:47.693001    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:47.808373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:47.808341    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:47.820453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:47.820425    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b"]
Oct 29 23:13:47.825350 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:47.825328    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b"
Oct 29 23:13:47.846818 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:47.846780    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b"]
Oct 29 23:13:47.852166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:47.852136    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-52f67b\""
Oct 29 23:13:47.852328 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:47.852213    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-52f67b-dockercfg-tbz9s\""
Oct 29 23:13:48.129884 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:48.021127    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526c46bd-79b3-446f-8b63-7c341d98b102-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b\" (UID: \"526c46bd-79b3-446f-8b63-7c341d98b102\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b"
Oct 29 23:13:48.137839 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:48.130207    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmr7b\" (UniqueName: \"kubernetes.io/projected/526c46bd-79b3-446f-8b63-7c341d98b102-kube-api-access-fmr7b\") pod \"payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b\" (UID: \"526c46bd-79b3-446f-8b63-7c341d98b102\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b"
Oct 29 23:13:48.219967 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:48.219927    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 29 23:13:48.220157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:48.219986    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf" podUID="2e3ed106-1a9a-43e6-998f-52b37686da76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 29 23:13:48.241557 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:48.241491    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526c46bd-79b3-446f-8b63-7c341d98b102-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b\" (UID: \"526c46bd-79b3-446f-8b63-7c341d98b102\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b"
Oct 29 23:13:48.241919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:48.241885    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmr7b\" (UniqueName: \"kubernetes.io/projected/526c46bd-79b3-446f-8b63-7c341d98b102-kube-api-access-fmr7b\") pod \"payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b\" (UID: \"526c46bd-79b3-446f-8b63-7c341d98b102\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b"
Oct 29 23:13:48.242388 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:48.242355    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526c46bd-79b3-446f-8b63-7c341d98b102-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b\" (UID: \"526c46bd-79b3-446f-8b63-7c341d98b102\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b"
Oct 29 23:13:48.298222 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:48.298177    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmr7b\" (UniqueName: \"kubernetes.io/projected/526c46bd-79b3-446f-8b63-7c341d98b102-kube-api-access-fmr7b\") pod \"payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b\" (UID: \"526c46bd-79b3-446f-8b63-7c341d98b102\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b"
Oct 29 23:13:48.440697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:48.440590    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b"
Oct 29 23:13:48.789552 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:13:48.789019    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526c46bd_79b3_446f_8b63_7c341d98b102.slice/crio-a184754d13eda23995fc614ebbc3ad9783bb3a27c19b1d5d691765e3a0d40f33 WatchSource:0}: Error finding container a184754d13eda23995fc614ebbc3ad9783bb3a27c19b1d5d691765e3a0d40f33: Status 404 returned error can't find the container with id a184754d13eda23995fc614ebbc3ad9783bb3a27c19b1d5d691765e3a0d40f33
Oct 29 23:13:48.796936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:48.796865    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b"]
Oct 29 23:13:49.219295 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:49.219260    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 29 23:13:49.219468 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:49.219339    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf" podUID="2e3ed106-1a9a-43e6-998f-52b37686da76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 29 23:13:49.386341 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:49.386303    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:49.645942 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:49.645902    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b" event={"ID":"526c46bd-79b3-446f-8b63-7c341d98b102","Type":"ContainerStarted","Data":"6d109caddac23d6fae3e37515aca77940ce1f4e1980ed70a463c741b99940a02"}
Oct 29 23:13:49.646135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:49.645971    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b" event={"ID":"526c46bd-79b3-446f-8b63-7c341d98b102","Type":"ContainerStarted","Data":"a184754d13eda23995fc614ebbc3ad9783bb3a27c19b1d5d691765e3a0d40f33"}
Oct 29 23:13:49.646198 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:49.646181    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b"
Oct 29 23:13:49.696699 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:49.696643    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b" podStartSLOduration=2.6966280769999997 podStartE2EDuration="2.696628077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:49.694424336 +0000 UTC m=+1833.037373082" watchObservedRunningTime="2026-10-29 23:13:49.696628077 +0000 UTC m=+1833.039576821"
Oct 29 23:13:50.219886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:50.219856    5428 patch_prober.go:29] interesting pod/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 29 23:13:50.220078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:50.219913    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf" podUID="2e3ed106-1a9a-43e6-998f-52b37686da76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 29 23:13:51.434494 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:51.434459    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:51.435673 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:51.435615    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:51.436633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:51.436610    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:13:51.748243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:51.748154    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:54.021823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:54.013423    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:54.175009 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:54.174980    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:55.253383 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:55.253350    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:55.742001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:55.741959    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:56.064980 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.064896    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"]
Oct 29 23:13:56.071064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.071034    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"
Oct 29 23:13:56.092384 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.092352    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"]
Oct 29 23:13:56.098997 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.098967    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-4ad17e-serving-cert\""
Oct 29 23:13:56.178288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.178251    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:56.198079 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.198046    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb\" (UID: \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"
Oct 29 23:13:56.198238 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.198105    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htj79\" (UniqueName: \"kubernetes.io/projected/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-kube-api-access-htj79\") pod \"maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb\" (UID: \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"
Oct 29 23:13:56.198291 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.198234    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb\" (UID: \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"
Oct 29 23:13:56.198326 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.198299    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb\" (UID: \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"
Oct 29 23:13:56.299893 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.299854    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb\" (UID: \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"
Oct 29 23:13:56.300305 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.299910    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb\" (UID: \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"
Oct 29 23:13:56.300305 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.300205    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb\" (UID: \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"
Oct 29 23:13:56.300305 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.300248    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htj79\" (UniqueName: \"kubernetes.io/projected/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-kube-api-access-htj79\") pod \"maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb\" (UID: \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"
Oct 29 23:13:56.300556 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.300531    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb\" (UID: \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"
Oct 29 23:13:56.302627 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.302594    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb\" (UID: \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"
Oct 29 23:13:56.302817 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.302784    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb\" (UID: \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"
Oct 29 23:13:56.302994 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.302975    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:56.324202 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.324137    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htj79\" (UniqueName: \"kubernetes.io/projected/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-kube-api-access-htj79\") pod \"maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb\" (UID: \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"
Oct 29 23:13:56.400193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.400155    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"
Oct 29 23:13:56.443008 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.442188    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:56.782675 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.782646    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"]
Oct 29 23:13:56.941431 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:56.941373    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:57.723175 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:57.723022    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb" event={"ID":"980d539c-9fe0-43d8-bff8-fcdb9aa963d9","Type":"ContainerStarted","Data":"4326986f2e356b8285a2fe992546e1886eb1297f5d7be1039caf041090d44214"}
Oct 29 23:13:57.723175 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:57.723090    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb" event={"ID":"980d539c-9fe0-43d8-bff8-fcdb9aa963d9","Type":"ContainerStarted","Data":"ed9e5f0a274b2f3c19093a262071e2a398fe13b1e64efb37c4ace48be550fdb7"}
Oct 29 23:13:57.723175 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:57.723137    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"
Oct 29 23:13:57.769045 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:57.768995    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb" podStartSLOduration=1.40327621 podStartE2EDuration="1.768981377s" totalImagesPullingTime="365.705167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:57.767276294 +0000 UTC m=+1841.110225038" watchObservedRunningTime="2026-10-29 23:13:57.768981377 +0000 UTC m=+1841.111930125"
Oct 29 23:13:58.302499 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:13:58.302461    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:00.203334 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:00.203297    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:00.353737 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:00.353689    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:00.546896 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:00.543195    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:00.657050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:00.657023    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b"
Oct 29 23:14:01.910371 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:01.910333    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:02.037384 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:02.037348    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:02.201717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:02.201619    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2"]
Oct 29 23:14:02.222161 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:02.222134    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2"
Oct 29 23:14:02.238390 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:02.237639    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-8a8b61\""
Oct 29 23:14:02.239925 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:02.238908    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-8a8b61-dockercfg-5574v\""
Oct 29 23:14:02.296559 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:02.296524    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2"]
Oct 29 23:14:02.297905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:02.297838    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wwdl\" (UniqueName: \"kubernetes.io/projected/c04a4a59-e364-41e3-bf19-cb3534ad9ec8-kube-api-access-5wwdl\") pod \"payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2\" (UID: \"c04a4a59-e364-41e3-bf19-cb3534ad9ec8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2"
Oct 29 23:14:02.311701 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:02.311668    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04a4a59-e364-41e3-bf19-cb3534ad9ec8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2\" (UID: \"c04a4a59-e364-41e3-bf19-cb3534ad9ec8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2"
Oct 29 23:14:02.414511 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:02.414472    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04a4a59-e364-41e3-bf19-cb3534ad9ec8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2\" (UID: \"c04a4a59-e364-41e3-bf19-cb3534ad9ec8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2"
Oct 29 23:14:02.414687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:02.414637    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wwdl\" (UniqueName: \"kubernetes.io/projected/c04a4a59-e364-41e3-bf19-cb3534ad9ec8-kube-api-access-5wwdl\") pod \"payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2\" (UID: \"c04a4a59-e364-41e3-bf19-cb3534ad9ec8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2"
Oct 29 23:14:02.415473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:02.415447    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04a4a59-e364-41e3-bf19-cb3534ad9ec8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2\" (UID: \"c04a4a59-e364-41e3-bf19-cb3534ad9ec8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2"
Oct 29 23:14:02.438926 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:02.438893    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wwdl\" (UniqueName: \"kubernetes.io/projected/c04a4a59-e364-41e3-bf19-cb3534ad9ec8-kube-api-access-5wwdl\") pod \"payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2\" (UID: \"c04a4a59-e364-41e3-bf19-cb3534ad9ec8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2"
Oct 29 23:14:02.554010 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:02.553918    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2"
Oct 29 23:14:02.870155 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:02.870128    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2"]
Oct 29 23:14:02.870635 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:14:02.870611    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04a4a59_e364_41e3_bf19_cb3534ad9ec8.slice/crio-fddac037f81293cacb403ae6fbeef17d62551155408f0663009cb2ab92ec985b WatchSource:0}: Error finding container fddac037f81293cacb403ae6fbeef17d62551155408f0663009cb2ab92ec985b: Status 404 returned error can't find the container with id fddac037f81293cacb403ae6fbeef17d62551155408f0663009cb2ab92ec985b
Oct 29 23:14:03.085498 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:03.085461    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:03.229645 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:03.229567    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:03.346891 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:03.346856    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:03.736152 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:03.736113    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"
Oct 29 23:14:03.775512 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:03.775474    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2" event={"ID":"c04a4a59-e364-41e3-bf19-cb3534ad9ec8","Type":"ContainerStarted","Data":"fde35d2819919cef358962e344e972fde9ca5b3ad1869d9a6b718e3307e9a2d6"}
Oct 29 23:14:03.775677 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:03.775525    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2" event={"ID":"c04a4a59-e364-41e3-bf19-cb3534ad9ec8","Type":"ContainerStarted","Data":"fddac037f81293cacb403ae6fbeef17d62551155408f0663009cb2ab92ec985b"}
Oct 29 23:14:03.775677 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:03.775550    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2"
Oct 29 23:14:03.832311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:03.832241    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2" podStartSLOduration=1.8322206859999999 podStartE2EDuration="1.832220686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:03.826189037 +0000 UTC m=+1847.169137784" watchObservedRunningTime="2026-10-29 23:14:03.832220686 +0000 UTC m=+1847.175169432"
Oct 29 23:14:05.050055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:05.049936    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:05.165299 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:05.165265    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:05.802654 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:05.802615    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:07.022920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:07.022887    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:08.805865 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:08.805828    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:10.608995 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:10.608949    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:12.284650 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:12.284610    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:13.372043 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:13.372005    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:13.577900 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:13.575315    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:14.457742 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:14.457712    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:14.787194 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:14.787116    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2"
Oct 29 23:14:16.263985 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:16.263956    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:17.101977 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:17.101941    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:17.370974 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:14:17.370937    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:14:18.293465 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:18.293434    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:20.170237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:20.170199    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:21.262282 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.262238    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"]
Oct 29 23:14:21.268669 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.268636    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.312378 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.312342    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"]
Oct 29 23:14:21.357244 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.357210    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-workload-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.357429 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.357387    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-token\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.357429 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.357424    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l598v\" (UniqueName: \"kubernetes.io/projected/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-kube-api-access-l598v\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.357552 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.357534    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-podinfo\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.357591 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.357563    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-credential-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.357591 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.357586    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-envoy\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.357683 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.357673    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-data\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.357725 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.357695    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istiod-ca-cert\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.357771 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.357735    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-workload-certs\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.459567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.459535    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-token\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.459762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.459598    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l598v\" (UniqueName: \"kubernetes.io/projected/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-kube-api-access-l598v\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.459762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.459678    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-podinfo\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.459762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.459722    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-credential-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.459762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.459759    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-envoy\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.460018 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.459820    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-data\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.460018 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.459852    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istiod-ca-cert\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.460018 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.459916    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-workload-certs\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.460018 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.459965    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-workload-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.460236 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.460190    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-credential-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.460396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.460373    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-workload-certs\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.460480 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.460436    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-workload-socket\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.460665 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.460639    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istiod-ca-cert\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.460779 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.460762    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-data\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.462112 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.462090    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-envoy\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.462233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.462216    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-podinfo\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.483115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.483084    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-token\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.483374 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.483350    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l598v\" (UniqueName: \"kubernetes.io/projected/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-kube-api-access-l598v\") pod \"e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") " pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.587611 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.587534    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:21.858289 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.858213    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:21.924382 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.924358    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"]
Oct 29 23:14:21.926914 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:14:21.926881    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a372a69_eaeb_4f73_ba14_fb4fa70a9ee6.slice/crio-ba6185ac36c0b2709efffc885125b03ec291c93a917631d3ce0345ebeb011470 WatchSource:0}: Error finding container ba6185ac36c0b2709efffc885125b03ec291c93a917631d3ce0345ebeb011470: Status 404 returned error can't find the container with id ba6185ac36c0b2709efffc885125b03ec291c93a917631d3ce0345ebeb011470
Oct 29 23:14:21.930031 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.929996    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:14:21.930123 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.930067    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:14:21.930123 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.930099    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:14:21.937559 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.937527    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8" event={"ID":"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6","Type":"ContainerStarted","Data":"ba6185ac36c0b2709efffc885125b03ec291c93a917631d3ce0345ebeb011470"}
Oct 29 23:14:21.979821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:21.979778    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:22.214180 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:22.214150    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"]
Oct 29 23:14:22.214376 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:22.214353    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2" podUID="9d3977ef-edc8-47f1-9779-c0f257e0e475" containerName="istio-proxy" containerID="cri-o://9f4ce7755f4e6ac925fc9e59641164798bbfd8e42914af5a0e0ab59af2653b86" gracePeriod=30
Oct 29 23:14:22.946867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:22.946830    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8" event={"ID":"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6","Type":"ContainerStarted","Data":"36c14d83dd3bacd8553d0b4a458b618fb9a58b21b7028e03a267357f1882e8bc"}
Oct 29 23:14:23.006260 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:23.006188    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8" podStartSLOduration=2.006167136 podStartE2EDuration="2.006167136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:23.003098945 +0000 UTC m=+1866.346047719" watchObservedRunningTime="2026-10-29 23:14:23.006167136 +0000 UTC m=+1866.349115881"
Oct 29 23:14:23.594730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:23.594696    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:23.596681 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:23.595049    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 29 23:14:23.596681 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:23.595104    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8" podUID="5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 29 23:14:23.870889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:23.867899    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:23.984902 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:23.984862    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:24.098679 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:24.098638    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:24.588681 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:24.588636    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 29 23:14:24.588918 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:24.588708    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8" podUID="5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 29 23:14:25.585283 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:25.585249    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:25.588398 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:25.588362    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 29 23:14:25.588562 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:25.588410    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8" podUID="5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 29 23:14:26.588474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:26.588438    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 29 23:14:26.588918 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:26.588499    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8" podUID="5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 29 23:14:27.280330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.280294    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:27.318954 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.318919    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-d9nr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 29 23:14:27.319231 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.319203    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2" podUID="9d3977ef-edc8-47f1-9779-c0f257e0e475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 29 23:14:27.588473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.588442    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 29 23:14:27.588625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.588511    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8" podUID="5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 29 23:14:27.594419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.594398    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 23:14:27.705464 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.705430    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d3977ef-edc8-47f1-9779-c0f257e0e475-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3977ef-edc8-47f1-9779-c0f257e0e475-istiod-ca-cert\") pod \"9d3977ef-edc8-47f1-9779-c0f257e0e475\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") "
Oct 29 23:14:27.705653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.705507    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-podinfo\") pod \"9d3977ef-edc8-47f1-9779-c0f257e0e475\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") "
Oct 29 23:14:27.705653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.705622    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3977ef-edc8-47f1-9779-c0f257e0e475-kube-api-access-mkl9k\" (UniqueName: \"kubernetes.io/projected/9d3977ef-edc8-47f1-9779-c0f257e0e475-kube-api-access-mkl9k\") pod \"9d3977ef-edc8-47f1-9779-c0f257e0e475\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") "
Oct 29 23:14:27.705821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.705654    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-credential-socket\") pod \"9d3977ef-edc8-47f1-9779-c0f257e0e475\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") "
Oct 29 23:14:27.705821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.705708    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-workload-certs\") pod \"9d3977ef-edc8-47f1-9779-c0f257e0e475\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") "
Oct 29 23:14:27.705821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.705751    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-data\") pod \"9d3977ef-edc8-47f1-9779-c0f257e0e475\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") "
Oct 29 23:14:27.706011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.705871    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-envoy\") pod \"9d3977ef-edc8-47f1-9779-c0f257e0e475\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") "
Oct 29 23:14:27.706011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.705900    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-workload-socket\") pod \"9d3977ef-edc8-47f1-9779-c0f257e0e475\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") "
Oct 29 23:14:27.706011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.705936    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-token\" (UniqueName: \"kubernetes.io/projected/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-token\") pod \"9d3977ef-edc8-47f1-9779-c0f257e0e475\" (UID: \"9d3977ef-edc8-47f1-9779-c0f257e0e475\") "
Oct 29 23:14:27.707401 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.707372    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3977ef-edc8-47f1-9779-c0f257e0e475-istiod-ca-cert" pod "9d3977ef-edc8-47f1-9779-c0f257e0e475" (UID: "9d3977ef-edc8-47f1-9779-c0f257e0e475"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:14:27.707572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.707546    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-credential-socket" pod "9d3977ef-edc8-47f1-9779-c0f257e0e475" (UID: "9d3977ef-edc8-47f1-9779-c0f257e0e475"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:14:27.707651 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.707610    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-data" pod "9d3977ef-edc8-47f1-9779-c0f257e0e475" (UID: "9d3977ef-edc8-47f1-9779-c0f257e0e475"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:14:27.707704 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.707689    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-workload-socket" pod "9d3977ef-edc8-47f1-9779-c0f257e0e475" (UID: "9d3977ef-edc8-47f1-9779-c0f257e0e475"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:14:27.707902 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.707877    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-workload-certs" pod "9d3977ef-edc8-47f1-9779-c0f257e0e475" (UID: "9d3977ef-edc8-47f1-9779-c0f257e0e475"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:14:27.712123 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.712097    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-token" pod "9d3977ef-edc8-47f1-9779-c0f257e0e475" (UID: "9d3977ef-edc8-47f1-9779-c0f257e0e475"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:14:27.723249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.722915    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3977ef-edc8-47f1-9779-c0f257e0e475-kube-api-access-mkl9k" pod "9d3977ef-edc8-47f1-9779-c0f257e0e475" (UID: "9d3977ef-edc8-47f1-9779-c0f257e0e475"). InnerVolumeSpecName "kube-api-access-mkl9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:14:27.723249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.723179    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-envoy" pod "9d3977ef-edc8-47f1-9779-c0f257e0e475" (UID: "9d3977ef-edc8-47f1-9779-c0f257e0e475"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:14:27.723464 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.723433    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-podinfo" pod "9d3977ef-edc8-47f1-9779-c0f257e0e475" (UID: "9d3977ef-edc8-47f1-9779-c0f257e0e475"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:14:27.808440 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.808343    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkl9k\" (UniqueName: \"kubernetes.io/projected/9d3977ef-edc8-47f1-9779-c0f257e0e475-kube-api-access-mkl9k\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:14:27.808440 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.808376    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:14:27.808440 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.808390    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:14:27.808440 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.808402    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:14:27.808440 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.808424    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:14:27.808440 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.808436    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d3977ef-edc8-47f1-9779-c0f257e0e475-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:14:27.808765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.808448    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:14:27.808765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.808460    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d3977ef-edc8-47f1-9779-c0f257e0e475-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:14:27.808765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:27.808470    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d3977ef-edc8-47f1-9779-c0f257e0e475-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:14:28.014767 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:28.014728    5428 generic.go:356] "Generic (PLEG): container finished" podID="9d3977ef-edc8-47f1-9779-c0f257e0e475" containerID="9f4ce7755f4e6ac925fc9e59641164798bbfd8e42914af5a0e0ab59af2653b86" exitCode=0
Oct 29 23:14:28.015003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:28.014782    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"
Oct 29 23:14:28.015003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:28.014833    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2" event={"ID":"9d3977ef-edc8-47f1-9779-c0f257e0e475","Type":"ContainerDied","Data":"9f4ce7755f4e6ac925fc9e59641164798bbfd8e42914af5a0e0ab59af2653b86"}
Oct 29 23:14:28.015003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:28.014889    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2" event={"ID":"9d3977ef-edc8-47f1-9779-c0f257e0e475","Type":"ContainerDied","Data":"65204e703f0ce4b82c99e572eaca96df874b5f27cb9504a16ce13430cdc4781f"}
Oct 29 23:14:28.015003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:28.014919    5428 scope.go:122] "RemoveContainer" containerID="9f4ce7755f4e6ac925fc9e59641164798bbfd8e42914af5a0e0ab59af2653b86"
Oct 29 23:14:28.028686 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:28.028663    5428 scope.go:122] "RemoveContainer" containerID="9f4ce7755f4e6ac925fc9e59641164798bbfd8e42914af5a0e0ab59af2653b86"
Oct 29 23:14:28.029003 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:14:28.028982    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f4ce7755f4e6ac925fc9e59641164798bbfd8e42914af5a0e0ab59af2653b86\": container with ID starting with 9f4ce7755f4e6ac925fc9e59641164798bbfd8e42914af5a0e0ab59af2653b86 not found: ID does not exist" containerID="9f4ce7755f4e6ac925fc9e59641164798bbfd8e42914af5a0e0ab59af2653b86"
Oct 29 23:14:28.029076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:28.029014    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f4ce7755f4e6ac925fc9e59641164798bbfd8e42914af5a0e0ab59af2653b86"} err="failed to get container status \"9f4ce7755f4e6ac925fc9e59641164798bbfd8e42914af5a0e0ab59af2653b86\": rpc error: code = NotFound desc = could not find container \"9f4ce7755f4e6ac925fc9e59641164798bbfd8e42914af5a0e0ab59af2653b86\": container with ID starting with 9f4ce7755f4e6ac925fc9e59641164798bbfd8e42914af5a0e0ab59af2653b86 not found: ID does not exist"
Oct 29 23:14:28.041644 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:28.041621    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:28.089214 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:28.089122    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"]
Oct 29 23:14:28.096216 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:28.096148    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-d9nr2"]
Oct 29 23:14:28.593613 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:28.593584    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:28.594079 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:28.594065    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:28.594645 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:28.594628    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:14:28.635682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:28.635632    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:28.779333 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:28.779302    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d3977ef-edc8-47f1-9779-c0f257e0e475" path="/var/lib/kubelet/pods/9d3977ef-edc8-47f1-9779-c0f257e0e475/volumes"
Oct 29 23:14:29.068996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:29.068487    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:29.268062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:29.268031    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:31.159540 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:31.151015    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:31.285831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:31.282922    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:33.472019 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:33.471971    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:34.930786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:34.930748    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:35.441247 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.441218    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"]
Oct 29 23:14:35.441889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.441861    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:35.442111 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.442094    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9d3977ef-edc8-47f1-9779-c0f257e0e475" containerName="istio-proxy"
Oct 29 23:14:35.442160 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.442113    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3977ef-edc8-47f1-9779-c0f257e0e475" containerName="istio-proxy"
Oct 29 23:14:35.442278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.442268    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3977ef-edc8-47f1-9779-c0f257e0e475" containerName="istio-proxy"
Oct 29 23:14:35.447959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.447939    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:14:35.488650 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.488617    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"]
Oct 29 23:14:35.520825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.508092    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-8a8b61-serving-cert\""
Oct 29 23:14:35.587824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.587184    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x\" (UID: \"d1c40c21-ebcf-4978-bc62-928061baf9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:14:35.587824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.587320    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrwjz\" (UniqueName: \"kubernetes.io/projected/d1c40c21-ebcf-4978-bc62-928061baf9b2-kube-api-access-hrwjz\") pod \"maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x\" (UID: \"d1c40c21-ebcf-4978-bc62-928061baf9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:14:35.587824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.587432    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c40c21-ebcf-4978-bc62-928061baf9b2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x\" (UID: \"d1c40c21-ebcf-4978-bc62-928061baf9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:14:35.587824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.587455    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x\" (UID: \"d1c40c21-ebcf-4978-bc62-928061baf9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:14:35.588340 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.588244    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:35.700682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.696685    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrwjz\" (UniqueName: \"kubernetes.io/projected/d1c40c21-ebcf-4978-bc62-928061baf9b2-kube-api-access-hrwjz\") pod \"maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x\" (UID: \"d1c40c21-ebcf-4978-bc62-928061baf9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:14:35.700682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.696871    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c40c21-ebcf-4978-bc62-928061baf9b2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x\" (UID: \"d1c40c21-ebcf-4978-bc62-928061baf9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:14:35.700682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.696905    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x\" (UID: \"d1c40c21-ebcf-4978-bc62-928061baf9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:14:35.700682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.697118    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x\" (UID: \"d1c40c21-ebcf-4978-bc62-928061baf9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:14:35.700682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.700574    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c40c21-ebcf-4978-bc62-928061baf9b2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x\" (UID: \"d1c40c21-ebcf-4978-bc62-928061baf9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:14:35.701195 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:14:35.700999    5428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-serving-cert: secret "maas-api-e2e-shared-b-w1-8a8b61-serving-cert" not found
Oct 29 23:14:35.701195 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:14:35.701062    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-maas-api-tls podName:d1c40c21-ebcf-4978-bc62-928061baf9b2 nodeName:}" failed. No retries permitted until 2026-10-29 23:14:36.201043697 +0000 UTC m=+1879.543992434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-maas-api-tls") pod "maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x" (UID: "d1c40c21-ebcf-4978-bc62-928061baf9b2") : secret "maas-api-e2e-shared-b-w1-8a8b61-serving-cert" not found
Oct 29 23:14:35.707288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.707258    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x\" (UID: \"d1c40c21-ebcf-4978-bc62-928061baf9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:14:35.743593 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.743561    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrwjz\" (UniqueName: \"kubernetes.io/projected/d1c40c21-ebcf-4978-bc62-928061baf9b2-kube-api-access-hrwjz\") pod \"maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x\" (UID: \"d1c40c21-ebcf-4978-bc62-928061baf9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:14:35.843160 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:35.843120    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:36.205269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:36.205230    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x\" (UID: \"d1c40c21-ebcf-4978-bc62-928061baf9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:14:36.205717 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:14:36.205377    5428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-serving-cert: secret "maas-api-e2e-shared-b-w1-8a8b61-serving-cert" not found
Oct 29 23:14:36.205717 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:14:36.205466    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-maas-api-tls podName:d1c40c21-ebcf-4978-bc62-928061baf9b2 nodeName:}" failed. No retries permitted until 2026-10-29 23:14:37.205452157 +0000 UTC m=+1880.548400892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-maas-api-tls") pod "maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x" (UID: "d1c40c21-ebcf-4978-bc62-928061baf9b2") : secret "maas-api-e2e-shared-b-w1-8a8b61-serving-cert" not found
Oct 29 23:14:36.718827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:36.718768    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:36.975847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:36.975748    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:37.221134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:37.221095    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x\" (UID: \"d1c40c21-ebcf-4978-bc62-928061baf9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:14:37.223629 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:37.223603    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x\" (UID: \"d1c40c21-ebcf-4978-bc62-928061baf9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:14:37.290743 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:37.290645    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:14:37.610227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:37.610195    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"]
Oct 29 23:14:37.611946 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:14:37.611917    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c40c21_ebcf_4978_bc62_928061baf9b2.slice/crio-af5a0bdf6d5c03ec7465d0b369d3beea34dc0b12da5afdd825ceb1abc8b96846 WatchSource:0}: Error finding container af5a0bdf6d5c03ec7465d0b369d3beea34dc0b12da5afdd825ceb1abc8b96846: Status 404 returned error can't find the container with id af5a0bdf6d5c03ec7465d0b369d3beea34dc0b12da5afdd825ceb1abc8b96846
Oct 29 23:14:37.837324 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:37.837295    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:38.101344 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:38.101287    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x" event={"ID":"d1c40c21-ebcf-4978-bc62-928061baf9b2","Type":"ContainerStarted","Data":"af5a0bdf6d5c03ec7465d0b369d3beea34dc0b12da5afdd825ceb1abc8b96846"}
Oct 29 23:14:38.338110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:38.338033    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:39.110905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:39.110865    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x" event={"ID":"d1c40c21-ebcf-4978-bc62-928061baf9b2","Type":"ContainerStarted","Data":"c57c51e438bf3bd37673c5dd28e26d6ef11bd14610734f026b954de6db2356c9"}
Oct 29 23:14:39.111092 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:39.110955    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:14:39.155052 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:39.155005    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x" podStartSLOduration=3.708202662 podStartE2EDuration="4.154991198s" totalImagesPullingTime="446.788536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:39.153711753 +0000 UTC m=+1882.496660497" watchObservedRunningTime="2026-10-29 23:14:39.154991198 +0000 UTC m=+1882.497939942"
Oct 29 23:14:39.776125 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:39.776085    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:39.903641 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:39.903603    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:40.273343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:40.273311    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:40.484493 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:40.484461    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:41.977121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:41.977088    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:42.240781 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:42.240690    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:42.341587 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:42.341556    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6"]
Oct 29 23:14:42.349820 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:42.349782    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6"]
Oct 29 23:14:42.350097 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:42.349913    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6"
Oct 29 23:14:42.364957 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:42.364932    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-2591f6\""
Oct 29 23:14:42.460853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:42.454122    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-2591f6-dockercfg-zx5bg\""
Oct 29 23:14:42.560764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:42.560674    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5nxd\" (UniqueName: \"kubernetes.io/projected/85670e7f-43f1-4f46-b691-88beee106a99-kube-api-access-z5nxd\") pod \"payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6\" (UID: \"85670e7f-43f1-4f46-b691-88beee106a99\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6"
Oct 29 23:14:42.560956 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:42.560935    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85670e7f-43f1-4f46-b691-88beee106a99-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6\" (UID: \"85670e7f-43f1-4f46-b691-88beee106a99\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6"
Oct 29 23:14:42.662194 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:42.662149    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5nxd\" (UniqueName: \"kubernetes.io/projected/85670e7f-43f1-4f46-b691-88beee106a99-kube-api-access-z5nxd\") pod \"payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6\" (UID: \"85670e7f-43f1-4f46-b691-88beee106a99\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6"
Oct 29 23:14:42.662396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:42.662379    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85670e7f-43f1-4f46-b691-88beee106a99-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6\" (UID: \"85670e7f-43f1-4f46-b691-88beee106a99\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6"
Oct 29 23:14:42.663224 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:42.663198    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85670e7f-43f1-4f46-b691-88beee106a99-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6\" (UID: \"85670e7f-43f1-4f46-b691-88beee106a99\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6"
Oct 29 23:14:42.685308 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:42.685277    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5nxd\" (UniqueName: \"kubernetes.io/projected/85670e7f-43f1-4f46-b691-88beee106a99-kube-api-access-z5nxd\") pod \"payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6\" (UID: \"85670e7f-43f1-4f46-b691-88beee106a99\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6"
Oct 29 23:14:42.967687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:42.967647    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6"
Oct 29 23:14:43.284104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:43.284073    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6"]
Oct 29 23:14:43.284536 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:14:43.284431    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85670e7f_43f1_4f46_b691_88beee106a99.slice/crio-a21e1b799c4852f8bc49c896ad573905a51c3c3abdd87d8b7175281a38cbfe19 WatchSource:0}: Error finding container a21e1b799c4852f8bc49c896ad573905a51c3c3abdd87d8b7175281a38cbfe19: Status 404 returned error can't find the container with id a21e1b799c4852f8bc49c896ad573905a51c3c3abdd87d8b7175281a38cbfe19
Oct 29 23:14:43.315081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:43.315059    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:44.155108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:44.155077    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6" event={"ID":"85670e7f-43f1-4f46-b691-88beee106a99","Type":"ContainerStarted","Data":"4f44d3e932ee51db2557695c5371a2000e660d58276d1429b7a331b148bd17e5"}
Oct 29 23:14:44.155293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:44.155129    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6" event={"ID":"85670e7f-43f1-4f46-b691-88beee106a99","Type":"ContainerStarted","Data":"a21e1b799c4852f8bc49c896ad573905a51c3c3abdd87d8b7175281a38cbfe19"}
Oct 29 23:14:44.155293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:44.155157    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6"
Oct 29 23:14:44.195585 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:44.195524    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6" podStartSLOduration=2.195506463 podStartE2EDuration="2.195506463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:44.194345545 +0000 UTC m=+1887.537294290" watchObservedRunningTime="2026-10-29 23:14:44.195506463 +0000 UTC m=+1887.538455209"
Oct 29 23:14:45.131685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:45.131654    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:14:45.519476 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:45.519390    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:47.112249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:47.112214    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:47.371878 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:14:47.371843    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:14:47.716543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:47.716450    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:47.942872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:47.942835    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:49.764924 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:49.764879    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:50.034879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:50.032434    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:51.740100 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:51.740066    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:53.727441 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:53.727411    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:55.168701 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:55.168668    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6"
Oct 29 23:14:55.382655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:55.382603    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:57.341461 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:57.341415    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:59.068316 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:14:59.068225    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:00.193938 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.193902    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj"]
Oct 29 23:15:00.201293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.201269    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj"
Oct 29 23:15:00.224441 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.224410    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv"]
Oct 29 23:15:00.230873 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.230844    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv"
Oct 29 23:15:00.263327 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.262991    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj"]
Oct 29 23:15:00.269863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.266989    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw"]
Oct 29 23:15:00.277960 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.277379    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r9nw\" (UniqueName: \"kubernetes.io/projected/7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8-kube-api-access-6r9nw\") pod \"maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv\" (UID: \"7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv"
Oct 29 23:15:00.277960 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.277505    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b5c2a2-7bce-4066-9390-7d2fac2998ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888595-ftwjj\" (UID: \"e9b5c2a2-7bce-4066-9390-7d2fac2998ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj"
Oct 29 23:15:00.277960 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.277724    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47n8m\" (UniqueName: \"kubernetes.io/projected/e9b5c2a2-7bce-4066-9390-7d2fac2998ab-kube-api-access-47n8m\") pod \"maas-api-key-cleanup-29888595-ftwjj\" (UID: \"e9b5c2a2-7bce-4066-9390-7d2fac2998ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj"
Oct 29 23:15:00.277960 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.277766    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv\" (UID: \"7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv"
Oct 29 23:15:00.280083 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.279867    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw"
Oct 29 23:15:00.280506 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.280161    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv"]
Oct 29 23:15:00.285869 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.285845    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\""
Oct 29 23:15:00.297521 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.291454    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9"]
Oct 29 23:15:00.300697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.300670    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw"]
Oct 29 23:15:00.300891 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.300784    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9"
Oct 29 23:15:00.303675 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.303653    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9"]
Oct 29 23:15:00.381737 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.381694    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b5c2a2-7bce-4066-9390-7d2fac2998ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888595-ftwjj\" (UID: \"e9b5c2a2-7bce-4066-9390-7d2fac2998ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj"
Oct 29 23:15:00.381964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.381780    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536f4758-d66b-4908-96df-222a7fc64517-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw\" (UID: \"536f4758-d66b-4908-96df-222a7fc64517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw"
Oct 29 23:15:00.381964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.381941    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47n8m\" (UniqueName: \"kubernetes.io/projected/e9b5c2a2-7bce-4066-9390-7d2fac2998ab-kube-api-access-47n8m\") pod \"maas-api-key-cleanup-29888595-ftwjj\" (UID: \"e9b5c2a2-7bce-4066-9390-7d2fac2998ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj"
Oct 29 23:15:00.382092 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.381984    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv\" (UID: \"7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv"
Oct 29 23:15:00.382148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.382096    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcwmf\" (UniqueName: \"kubernetes.io/projected/bd15453b-4398-4e27-a18b-0b44251a8aff-kube-api-access-mcwmf\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9\" (UID: \"bd15453b-4398-4e27-a18b-0b44251a8aff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9"
Oct 29 23:15:00.382148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.382135    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd15453b-4398-4e27-a18b-0b44251a8aff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9\" (UID: \"bd15453b-4398-4e27-a18b-0b44251a8aff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9"
Oct 29 23:15:00.382264 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.382199    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j52tn\" (UniqueName: \"kubernetes.io/projected/536f4758-d66b-4908-96df-222a7fc64517-kube-api-access-j52tn\") pod \"maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw\" (UID: \"536f4758-d66b-4908-96df-222a7fc64517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw"
Oct 29 23:15:00.382351 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.382333    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r9nw\" (UniqueName: \"kubernetes.io/projected/7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8-kube-api-access-6r9nw\") pod \"maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv\" (UID: \"7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv"
Oct 29 23:15:00.384134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.384100    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b5c2a2-7bce-4066-9390-7d2fac2998ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888595-ftwjj\" (UID: \"e9b5c2a2-7bce-4066-9390-7d2fac2998ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj"
Oct 29 23:15:00.385355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.385330    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv\" (UID: \"7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv"
Oct 29 23:15:00.405901 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.405872    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r9nw\" (UniqueName: \"kubernetes.io/projected/7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8-kube-api-access-6r9nw\") pod \"maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv\" (UID: \"7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv"
Oct 29 23:15:00.407543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.407516    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47n8m\" (UniqueName: \"kubernetes.io/projected/e9b5c2a2-7bce-4066-9390-7d2fac2998ab-kube-api-access-47n8m\") pod \"maas-api-key-cleanup-29888595-ftwjj\" (UID: \"e9b5c2a2-7bce-4066-9390-7d2fac2998ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj"
Oct 29 23:15:00.483844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.483740    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536f4758-d66b-4908-96df-222a7fc64517-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw\" (UID: \"536f4758-d66b-4908-96df-222a7fc64517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw"
Oct 29 23:15:00.484019 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.483997    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcwmf\" (UniqueName: \"kubernetes.io/projected/bd15453b-4398-4e27-a18b-0b44251a8aff-kube-api-access-mcwmf\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9\" (UID: \"bd15453b-4398-4e27-a18b-0b44251a8aff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9"
Oct 29 23:15:00.484095 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.484042    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd15453b-4398-4e27-a18b-0b44251a8aff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9\" (UID: \"bd15453b-4398-4e27-a18b-0b44251a8aff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9"
Oct 29 23:15:00.484155 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.484103    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j52tn\" (UniqueName: \"kubernetes.io/projected/536f4758-d66b-4908-96df-222a7fc64517-kube-api-access-j52tn\") pod \"maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw\" (UID: \"536f4758-d66b-4908-96df-222a7fc64517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw"
Oct 29 23:15:00.484374 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.484352    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536f4758-d66b-4908-96df-222a7fc64517-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw\" (UID: \"536f4758-d66b-4908-96df-222a7fc64517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw"
Oct 29 23:15:00.484733 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.484713    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd15453b-4398-4e27-a18b-0b44251a8aff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9\" (UID: \"bd15453b-4398-4e27-a18b-0b44251a8aff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9"
Oct 29 23:15:00.506995 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.506960    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j52tn\" (UniqueName: \"kubernetes.io/projected/536f4758-d66b-4908-96df-222a7fc64517-kube-api-access-j52tn\") pod \"maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw\" (UID: \"536f4758-d66b-4908-96df-222a7fc64517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw"
Oct 29 23:15:00.507128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.507015    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcwmf\" (UniqueName: \"kubernetes.io/projected/bd15453b-4398-4e27-a18b-0b44251a8aff-kube-api-access-mcwmf\") pod \"maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9\" (UID: \"bd15453b-4398-4e27-a18b-0b44251a8aff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9"
Oct 29 23:15:00.516925 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.516900    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj"
Oct 29 23:15:00.551446 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.551417    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv"
Oct 29 23:15:00.618826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.615690    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw"
Oct 29 23:15:00.632903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.632678    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9"
Oct 29 23:15:00.837138 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:15:00.837086    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b5c2a2_7bce_4066_9390_7d2fac2998ab.slice/crio-489b768ff7fe90a24fc83fcd8b7ffd110553044e8429d649110774c348147e7f WatchSource:0}: Error finding container 489b768ff7fe90a24fc83fcd8b7ffd110553044e8429d649110774c348147e7f: Status 404 returned error can't find the container with id 489b768ff7fe90a24fc83fcd8b7ffd110553044e8429d649110774c348147e7f
Oct 29 23:15:00.839781 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.839754    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj"]
Oct 29 23:15:00.885029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.884994    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv"]
Oct 29 23:15:00.888162 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:15:00.888128    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c90c88f_d7b3_44f7_9ebb_ea9226ba92f8.slice/crio-267a6e73abb43712e243a462f402afaf3540630da9fcd01700a844a485f6f4ca WatchSource:0}: Error finding container 267a6e73abb43712e243a462f402afaf3540630da9fcd01700a844a485f6f4ca: Status 404 returned error can't find the container with id 267a6e73abb43712e243a462f402afaf3540630da9fcd01700a844a485f6f4ca
Oct 29 23:15:00.891228 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.891188    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:00.966169 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.966140    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw"]
Oct 29 23:15:00.986754 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:00.986730    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9"]
Oct 29 23:15:01.063101 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:15:01.063067    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536f4758_d66b_4908_96df_222a7fc64517.slice/crio-53806aed65c74df6298637daf07e296208ed0c61894c6dddc5ee291e02fbd444 WatchSource:0}: Error finding container 53806aed65c74df6298637daf07e296208ed0c61894c6dddc5ee291e02fbd444: Status 404 returned error can't find the container with id 53806aed65c74df6298637daf07e296208ed0c61894c6dddc5ee291e02fbd444
Oct 29 23:15:01.063974 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:15:01.063906    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd15453b_4398_4e27_a18b_0b44251a8aff.slice/crio-5a0ad3a25edf3b268cdbabc3dd5cde9c57ed288e87125291029318ef13cfdf95 WatchSource:0}: Error finding container 5a0ad3a25edf3b268cdbabc3dd5cde9c57ed288e87125291029318ef13cfdf95: Status 404 returned error can't find the container with id 5a0ad3a25edf3b268cdbabc3dd5cde9c57ed288e87125291029318ef13cfdf95
Oct 29 23:15:01.315613 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:01.315580    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv" event={"ID":"7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8","Type":"ContainerStarted","Data":"267a6e73abb43712e243a462f402afaf3540630da9fcd01700a844a485f6f4ca"}
Oct 29 23:15:01.316973 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:01.316942    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw" event={"ID":"536f4758-d66b-4908-96df-222a7fc64517","Type":"ContainerStarted","Data":"53806aed65c74df6298637daf07e296208ed0c61894c6dddc5ee291e02fbd444"}
Oct 29 23:15:01.318281 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:01.318252    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj" event={"ID":"e9b5c2a2-7bce-4066-9390-7d2fac2998ab","Type":"ContainerStarted","Data":"489b768ff7fe90a24fc83fcd8b7ffd110553044e8429d649110774c348147e7f"}
Oct 29 23:15:01.319558 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:01.319533    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9" event={"ID":"bd15453b-4398-4e27-a18b-0b44251a8aff","Type":"ContainerStarted","Data":"5a0ad3a25edf3b268cdbabc3dd5cde9c57ed288e87125291029318ef13cfdf95"}
Oct 29 23:15:02.331114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:02.331021    5428 generic.go:356] "Generic (PLEG): container finished" podID="bd15453b-4398-4e27-a18b-0b44251a8aff" containerID="cb349bf3a3e8a1026ad38ba1f1849e56aab1b29216136ebd147ad9d715d408bd" exitCode=60
Oct 29 23:15:02.331114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:02.331100    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9" event={"ID":"bd15453b-4398-4e27-a18b-0b44251a8aff","Type":"ContainerDied","Data":"cb349bf3a3e8a1026ad38ba1f1849e56aab1b29216136ebd147ad9d715d408bd"}
Oct 29 23:15:02.331629 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:02.331441    5428 scope.go:122] "RemoveContainer" containerID="cb349bf3a3e8a1026ad38ba1f1849e56aab1b29216136ebd147ad9d715d408bd"
Oct 29 23:15:02.333000 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:02.332972    5428 generic.go:356] "Generic (PLEG): container finished" podID="7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8" containerID="e51a0b5a4f65f9b9659365dde8db86770cf15828c068912589679ccc39629693" exitCode=60
Oct 29 23:15:02.333111 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:02.333018    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv" event={"ID":"7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8","Type":"ContainerDied","Data":"e51a0b5a4f65f9b9659365dde8db86770cf15828c068912589679ccc39629693"}
Oct 29 23:15:02.333332 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:02.333316    5428 scope.go:122] "RemoveContainer" containerID="e51a0b5a4f65f9b9659365dde8db86770cf15828c068912589679ccc39629693"
Oct 29 23:15:02.335089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:02.334993    5428 generic.go:356] "Generic (PLEG): container finished" podID="536f4758-d66b-4908-96df-222a7fc64517" containerID="7d4a5c45645e352b0e287648f8326c5256061b368b8e0d1e32f2693d910cbaec" exitCode=60
Oct 29 23:15:02.335089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:02.335016    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw" event={"ID":"536f4758-d66b-4908-96df-222a7fc64517","Type":"ContainerDied","Data":"7d4a5c45645e352b0e287648f8326c5256061b368b8e0d1e32f2693d910cbaec"}
Oct 29 23:15:02.335374 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:02.335328    5428 scope.go:122] "RemoveContainer" containerID="7d4a5c45645e352b0e287648f8326c5256061b368b8e0d1e32f2693d910cbaec"
Oct 29 23:15:02.337055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:02.336884    5428 generic.go:356] "Generic (PLEG): container finished" podID="e9b5c2a2-7bce-4066-9390-7d2fac2998ab" containerID="6bc198d4975b6e719aa838bf91a3c27e71d69f664b3661fb0d80a31d77f14e9b" exitCode=60
Oct 29 23:15:02.337055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:02.336914    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj" event={"ID":"e9b5c2a2-7bce-4066-9390-7d2fac2998ab","Type":"ContainerDied","Data":"6bc198d4975b6e719aa838bf91a3c27e71d69f664b3661fb0d80a31d77f14e9b"}
Oct 29 23:15:02.337245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:02.337228    5428 scope.go:122] "RemoveContainer" containerID="6bc198d4975b6e719aa838bf91a3c27e71d69f664b3661fb0d80a31d77f14e9b"
Oct 29 23:15:03.148869 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:03.148757    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:03.346668 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:03.346632    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw" event={"ID":"536f4758-d66b-4908-96df-222a7fc64517","Type":"ContainerStarted","Data":"40aea99794cd3fdd7b6560e1982debc51fde8e3e7423ca7dd698266c3aa2e091"}
Oct 29 23:15:03.348872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:03.348838    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj" event={"ID":"e9b5c2a2-7bce-4066-9390-7d2fac2998ab","Type":"ContainerStarted","Data":"94a94eb6501e70b9cd42a4e3870c6ce1f259796b693daba51b7f7f7a0f0de01b"}
Oct 29 23:15:03.350982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:03.350953    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9" event={"ID":"bd15453b-4398-4e27-a18b-0b44251a8aff","Type":"ContainerStarted","Data":"7f04016cd9461ca926c060d8fadd803922ac077d2ebabcb1a863d8c1d1cadb7a"}
Oct 29 23:15:03.352866 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:03.352836    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv" event={"ID":"7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8","Type":"ContainerStarted","Data":"ba3d038c6a7ae5595eafc7e75580238245c17fcf6c8d401b14936647278a0403"}
Oct 29 23:15:03.387781 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:03.387740    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw" podStartSLOduration=3.387726266 podStartE2EDuration="3.387726266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.386348507 +0000 UTC m=+1906.729297254" watchObservedRunningTime="2026-10-29 23:15:03.387726266 +0000 UTC m=+1906.730675010"
Oct 29 23:15:03.429216 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:03.429127    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj" podStartSLOduration=3.429113153 podStartE2EDuration="3.429113153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.424962064 +0000 UTC m=+1906.767910809" watchObservedRunningTime="2026-10-29 23:15:03.429113153 +0000 UTC m=+1906.772061898"
Oct 29 23:15:03.466593 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:03.466550    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv" podStartSLOduration=3.466536289 podStartE2EDuration="3.466536289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.463556093 +0000 UTC m=+1906.806504838" watchObservedRunningTime="2026-10-29 23:15:03.466536289 +0000 UTC m=+1906.809485033"
Oct 29 23:15:03.505580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:03.505528    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9" podStartSLOduration=3.505512532 podStartE2EDuration="3.505512532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.501773235 +0000 UTC m=+1906.844721991" watchObservedRunningTime="2026-10-29 23:15:03.505512532 +0000 UTC m=+1906.848461277"
Oct 29 23:15:04.363812 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:04.363772    5428 generic.go:356] "Generic (PLEG): container finished" podID="536f4758-d66b-4908-96df-222a7fc64517" containerID="40aea99794cd3fdd7b6560e1982debc51fde8e3e7423ca7dd698266c3aa2e091" exitCode=60
Oct 29 23:15:04.364354 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:04.363839    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw" event={"ID":"536f4758-d66b-4908-96df-222a7fc64517","Type":"ContainerDied","Data":"40aea99794cd3fdd7b6560e1982debc51fde8e3e7423ca7dd698266c3aa2e091"}
Oct 29 23:15:04.364354 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:04.363912    5428 scope.go:122] "RemoveContainer" containerID="7d4a5c45645e352b0e287648f8326c5256061b368b8e0d1e32f2693d910cbaec"
Oct 29 23:15:04.364354 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:04.364249    5428 scope.go:122] "RemoveContainer" containerID="40aea99794cd3fdd7b6560e1982debc51fde8e3e7423ca7dd698266c3aa2e091"
Oct 29 23:15:04.364670 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:15:04.364649    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw_odh-ai-gateway-infra(536f4758-d66b-4908-96df-222a7fc64517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw" podUID="536f4758-d66b-4908-96df-222a7fc64517"
Oct 29 23:15:04.366154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:04.366123    5428 generic.go:356] "Generic (PLEG): container finished" podID="e9b5c2a2-7bce-4066-9390-7d2fac2998ab" containerID="94a94eb6501e70b9cd42a4e3870c6ce1f259796b693daba51b7f7f7a0f0de01b" exitCode=60
Oct 29 23:15:04.366261 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:04.366163    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj" event={"ID":"e9b5c2a2-7bce-4066-9390-7d2fac2998ab","Type":"ContainerDied","Data":"94a94eb6501e70b9cd42a4e3870c6ce1f259796b693daba51b7f7f7a0f0de01b"}
Oct 29 23:15:04.366545 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:04.366527    5428 scope.go:122] "RemoveContainer" containerID="94a94eb6501e70b9cd42a4e3870c6ce1f259796b693daba51b7f7f7a0f0de01b"
Oct 29 23:15:04.366954 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:15:04.366932    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888595-ftwjj_odh-ai-gateway-infra(e9b5c2a2-7bce-4066-9390-7d2fac2998ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj" podUID="e9b5c2a2-7bce-4066-9390-7d2fac2998ab"
Oct 29 23:15:04.368346 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:04.368312    5428 generic.go:356] "Generic (PLEG): container finished" podID="bd15453b-4398-4e27-a18b-0b44251a8aff" containerID="7f04016cd9461ca926c060d8fadd803922ac077d2ebabcb1a863d8c1d1cadb7a" exitCode=60
Oct 29 23:15:04.368470 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:04.368396    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9" event={"ID":"bd15453b-4398-4e27-a18b-0b44251a8aff","Type":"ContainerDied","Data":"7f04016cd9461ca926c060d8fadd803922ac077d2ebabcb1a863d8c1d1cadb7a"}
Oct 29 23:15:04.368687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:04.368670    5428 scope.go:122] "RemoveContainer" containerID="7f04016cd9461ca926c060d8fadd803922ac077d2ebabcb1a863d8c1d1cadb7a"
Oct 29 23:15:04.369039 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:15:04.369007    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9_odh-ai-gateway-infra(bd15453b-4398-4e27-a18b-0b44251a8aff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9" podUID="bd15453b-4398-4e27-a18b-0b44251a8aff"
Oct 29 23:15:04.370463 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:04.370435    5428 generic.go:356] "Generic (PLEG): container finished" podID="7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8" containerID="ba3d038c6a7ae5595eafc7e75580238245c17fcf6c8d401b14936647278a0403" exitCode=60
Oct 29 23:15:04.370551 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:04.370497    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv" event={"ID":"7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8","Type":"ContainerDied","Data":"ba3d038c6a7ae5595eafc7e75580238245c17fcf6c8d401b14936647278a0403"}
Oct 29 23:15:04.370717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:04.370705    5428 scope.go:122] "RemoveContainer" containerID="ba3d038c6a7ae5595eafc7e75580238245c17fcf6c8d401b14936647278a0403"
Oct 29 23:15:04.371016 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:15:04.370999    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv_odh-ai-gateway-infra(7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv" podUID="7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8"
Oct 29 23:15:04.378793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:04.378774    5428 scope.go:122] "RemoveContainer" containerID="6bc198d4975b6e719aa838bf91a3c27e71d69f664b3661fb0d80a31d77f14e9b"
Oct 29 23:15:04.393610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:04.393583    5428 scope.go:122] "RemoveContainer" containerID="cb349bf3a3e8a1026ad38ba1f1849e56aab1b29216136ebd147ad9d715d408bd"
Oct 29 23:15:04.408559 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:04.408512    5428 scope.go:122] "RemoveContainer" containerID="e51a0b5a4f65f9b9659365dde8db86770cf15828c068912589679ccc39629693"
Oct 29 23:15:05.014922 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:05.014883    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:05.381192 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:05.381163    5428 scope.go:122] "RemoveContainer" containerID="40aea99794cd3fdd7b6560e1982debc51fde8e3e7423ca7dd698266c3aa2e091"
Oct 29 23:15:05.381703 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:15:05.381534    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw_odh-ai-gateway-infra(536f4758-d66b-4908-96df-222a7fc64517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw" podUID="536f4758-d66b-4908-96df-222a7fc64517"
Oct 29 23:15:05.382987 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:05.382959    5428 scope.go:122] "RemoveContainer" containerID="94a94eb6501e70b9cd42a4e3870c6ce1f259796b693daba51b7f7f7a0f0de01b"
Oct 29 23:15:05.383312 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:15:05.383287    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888595-ftwjj_odh-ai-gateway-infra(e9b5c2a2-7bce-4066-9390-7d2fac2998ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj" podUID="e9b5c2a2-7bce-4066-9390-7d2fac2998ab"
Oct 29 23:15:05.384638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:05.384620    5428 scope.go:122] "RemoveContainer" containerID="7f04016cd9461ca926c060d8fadd803922ac077d2ebabcb1a863d8c1d1cadb7a"
Oct 29 23:15:05.384944 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:15:05.384918    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9_odh-ai-gateway-infra(bd15453b-4398-4e27-a18b-0b44251a8aff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9" podUID="bd15453b-4398-4e27-a18b-0b44251a8aff"
Oct 29 23:15:05.386579 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:05.386560    5428 scope.go:122] "RemoveContainer" containerID="ba3d038c6a7ae5595eafc7e75580238245c17fcf6c8d401b14936647278a0403"
Oct 29 23:15:05.386891 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:15:05.386869    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv_odh-ai-gateway-infra(7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv" podUID="7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8"
Oct 29 23:15:06.788986 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:06.788953    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:07.413287 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:07.413256    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:08.682964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:08.682925    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:10.367381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:10.367343    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:13.640266 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:13.640223    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:14.779248 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:14.779218    5428 scope.go:122] "RemoveContainer" containerID="94a94eb6501e70b9cd42a4e3870c6ce1f259796b693daba51b7f7f7a0f0de01b"
Oct 29 23:15:15.421248 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:15.421220    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:15.504200 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:15.504158    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj" event={"ID":"e9b5c2a2-7bce-4066-9390-7d2fac2998ab","Type":"ContainerStarted","Data":"dbfcb7239ea61a45c0436e85accc80f77ebac39a81e62adfe673b3f02c7644ca"}
Oct 29 23:15:15.769666 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:15.769636    5428 scope.go:122] "RemoveContainer" containerID="40aea99794cd3fdd7b6560e1982debc51fde8e3e7423ca7dd698266c3aa2e091"
Oct 29 23:15:16.514239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:16.514203    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw" event={"ID":"536f4758-d66b-4908-96df-222a7fc64517","Type":"ContainerStarted","Data":"26540de541a0de2c3f6056e9ae85a94086d31d5c41233366d0c426208dde361c"}
Oct 29 23:15:16.516325 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:16.516294    5428 generic.go:356] "Generic (PLEG): container finished" podID="e9b5c2a2-7bce-4066-9390-7d2fac2998ab" containerID="dbfcb7239ea61a45c0436e85accc80f77ebac39a81e62adfe673b3f02c7644ca" exitCode=60
Oct 29 23:15:16.516445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:16.516371    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj" event={"ID":"e9b5c2a2-7bce-4066-9390-7d2fac2998ab","Type":"ContainerDied","Data":"dbfcb7239ea61a45c0436e85accc80f77ebac39a81e62adfe673b3f02c7644ca"}
Oct 29 23:15:16.516445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:16.516433    5428 scope.go:122] "RemoveContainer" containerID="94a94eb6501e70b9cd42a4e3870c6ce1f259796b693daba51b7f7f7a0f0de01b"
Oct 29 23:15:16.516786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:16.516761    5428 scope.go:122] "RemoveContainer" containerID="dbfcb7239ea61a45c0436e85accc80f77ebac39a81e62adfe673b3f02c7644ca"
Oct 29 23:15:16.517124 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:15:16.517102    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888595-ftwjj_odh-ai-gateway-infra(e9b5c2a2-7bce-4066-9390-7d2fac2998ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj" podUID="e9b5c2a2-7bce-4066-9390-7d2fac2998ab"
Oct 29 23:15:16.557904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:16.557874    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj"]
Oct 29 23:15:16.781324 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:16.781243    5428 scope.go:122] "RemoveContainer" containerID="7f04016cd9461ca926c060d8fadd803922ac077d2ebabcb1a863d8c1d1cadb7a"
Oct 29 23:15:17.370215 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:15:17.370178    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:15:17.527323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:17.527220    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9" event={"ID":"bd15453b-4398-4e27-a18b-0b44251a8aff","Type":"ContainerStarted","Data":"fa1607b8b9e31a796af33a5eb2223bdd875a5a24e533b07d07311c06770a5d7b"}
Oct 29 23:15:17.529486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:17.529460    5428 generic.go:356] "Generic (PLEG): container finished" podID="536f4758-d66b-4908-96df-222a7fc64517" containerID="26540de541a0de2c3f6056e9ae85a94086d31d5c41233366d0c426208dde361c" exitCode=60
Oct 29 23:15:17.529644 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:17.529522    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw" event={"ID":"536f4758-d66b-4908-96df-222a7fc64517","Type":"ContainerDied","Data":"26540de541a0de2c3f6056e9ae85a94086d31d5c41233366d0c426208dde361c"}
Oct 29 23:15:17.529644 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:17.529573    5428 scope.go:122] "RemoveContainer" containerID="40aea99794cd3fdd7b6560e1982debc51fde8e3e7423ca7dd698266c3aa2e091"
Oct 29 23:15:17.530008 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:17.529982    5428 scope.go:122] "RemoveContainer" containerID="26540de541a0de2c3f6056e9ae85a94086d31d5c41233366d0c426208dde361c"
Oct 29 23:15:17.530387 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:15:17.530362    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw_odh-ai-gateway-infra(536f4758-d66b-4908-96df-222a7fc64517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw" podUID="536f4758-d66b-4908-96df-222a7fc64517"
Oct 29 23:15:17.575341 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:17.575087    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw"]
Oct 29 23:15:17.682947 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:17.681969    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:17.781467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:17.780860    5428 scope.go:122] "RemoveContainer" containerID="ba3d038c6a7ae5595eafc7e75580238245c17fcf6c8d401b14936647278a0403"
Oct 29 23:15:17.849210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:17.849184    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj"
Oct 29 23:15:18.006504 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.006470    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9b5c2a2-7bce-4066-9390-7d2fac2998ab-kube-api-access-47n8m\" (UniqueName: \"kubernetes.io/projected/e9b5c2a2-7bce-4066-9390-7d2fac2998ab-kube-api-access-47n8m\") pod \"e9b5c2a2-7bce-4066-9390-7d2fac2998ab\" (UID: \"e9b5c2a2-7bce-4066-9390-7d2fac2998ab\") "
Oct 29 23:15:18.006703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.006541    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9b5c2a2-7bce-4066-9390-7d2fac2998ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b5c2a2-7bce-4066-9390-7d2fac2998ab-maas-api-service-ca\") pod \"e9b5c2a2-7bce-4066-9390-7d2fac2998ab\" (UID: \"e9b5c2a2-7bce-4066-9390-7d2fac2998ab\") "
Oct 29 23:15:18.006923 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.006901    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9b5c2a2-7bce-4066-9390-7d2fac2998ab-maas-api-service-ca" pod "e9b5c2a2-7bce-4066-9390-7d2fac2998ab" (UID: "e9b5c2a2-7bce-4066-9390-7d2fac2998ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:18.007135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.007115    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b5c2a2-7bce-4066-9390-7d2fac2998ab-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:15:18.008559 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.008533    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b5c2a2-7bce-4066-9390-7d2fac2998ab-kube-api-access-47n8m" pod "e9b5c2a2-7bce-4066-9390-7d2fac2998ab" (UID: "e9b5c2a2-7bce-4066-9390-7d2fac2998ab"). InnerVolumeSpecName "kube-api-access-47n8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:18.108176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.108139    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47n8m\" (UniqueName: \"kubernetes.io/projected/e9b5c2a2-7bce-4066-9390-7d2fac2998ab-kube-api-access-47n8m\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:15:18.542879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.542842    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj" event={"ID":"e9b5c2a2-7bce-4066-9390-7d2fac2998ab","Type":"ContainerDied","Data":"489b768ff7fe90a24fc83fcd8b7ffd110553044e8429d649110774c348147e7f"}
Oct 29 23:15:18.543305 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.542911    5428 scope.go:122] "RemoveContainer" containerID="dbfcb7239ea61a45c0436e85accc80f77ebac39a81e62adfe673b3f02c7644ca"
Oct 29 23:15:18.543305 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.542852    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj"
Oct 29 23:15:18.545131 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.545105    5428 generic.go:356] "Generic (PLEG): container finished" podID="bd15453b-4398-4e27-a18b-0b44251a8aff" containerID="fa1607b8b9e31a796af33a5eb2223bdd875a5a24e533b07d07311c06770a5d7b" exitCode=60
Oct 29 23:15:18.545251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.545149    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9" event={"ID":"bd15453b-4398-4e27-a18b-0b44251a8aff","Type":"ContainerDied","Data":"fa1607b8b9e31a796af33a5eb2223bdd875a5a24e533b07d07311c06770a5d7b"}
Oct 29 23:15:18.545647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.545578    5428 scope.go:122] "RemoveContainer" containerID="fa1607b8b9e31a796af33a5eb2223bdd875a5a24e533b07d07311c06770a5d7b"
Oct 29 23:15:18.545951 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:15:18.545923    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9_odh-ai-gateway-infra(bd15453b-4398-4e27-a18b-0b44251a8aff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9" podUID="bd15453b-4398-4e27-a18b-0b44251a8aff"
Oct 29 23:15:18.550096 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.550070    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv" event={"ID":"7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8","Type":"ContainerStarted","Data":"bbd7f1a578fa0e066e8bffc51436d93488b6edb8dff9c9492a152b98ddc85666"}
Oct 29 23:15:18.555831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.555789    5428 scope.go:122] "RemoveContainer" containerID="7f04016cd9461ca926c060d8fadd803922ac077d2ebabcb1a863d8c1d1cadb7a"
Oct 29 23:15:18.581949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.581923    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9"]
Oct 29 23:15:18.696981 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.696953    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw"
Oct 29 23:15:18.713957 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.713927    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj"]
Oct 29 23:15:18.717383 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.717360    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888595-ftwjj"]
Oct 29 23:15:18.791876 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.788308    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9b5c2a2-7bce-4066-9390-7d2fac2998ab" path="/var/lib/kubelet/pods/e9b5c2a2-7bce-4066-9390-7d2fac2998ab/volumes"
Oct 29 23:15:18.819460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.819416    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536f4758-d66b-4908-96df-222a7fc64517-kube-api-access-j52tn\" (UniqueName: \"kubernetes.io/projected/536f4758-d66b-4908-96df-222a7fc64517-kube-api-access-j52tn\") pod \"536f4758-d66b-4908-96df-222a7fc64517\" (UID: \"536f4758-d66b-4908-96df-222a7fc64517\") "
Oct 29 23:15:18.824470 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.824431    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/536f4758-d66b-4908-96df-222a7fc64517-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536f4758-d66b-4908-96df-222a7fc64517-maas-api-service-ca\") pod \"536f4758-d66b-4908-96df-222a7fc64517\" (UID: \"536f4758-d66b-4908-96df-222a7fc64517\") "
Oct 29 23:15:18.825172 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.825129    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536f4758-d66b-4908-96df-222a7fc64517-maas-api-service-ca" pod "536f4758-d66b-4908-96df-222a7fc64517" (UID: "536f4758-d66b-4908-96df-222a7fc64517"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:18.827379 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.827351    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536f4758-d66b-4908-96df-222a7fc64517-kube-api-access-j52tn" pod "536f4758-d66b-4908-96df-222a7fc64517" (UID: "536f4758-d66b-4908-96df-222a7fc64517"). InnerVolumeSpecName "kube-api-access-j52tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:18.926887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.926854    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j52tn\" (UniqueName: \"kubernetes.io/projected/536f4758-d66b-4908-96df-222a7fc64517-kube-api-access-j52tn\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:15:18.926887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.926885    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536f4758-d66b-4908-96df-222a7fc64517-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:15:18.984116 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:18.984002    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:19.561247 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.561218    5428 generic.go:356] "Generic (PLEG): container finished" podID="7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8" containerID="bbd7f1a578fa0e066e8bffc51436d93488b6edb8dff9c9492a152b98ddc85666" exitCode=60
Oct 29 23:15:19.561694 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.561366    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv" event={"ID":"7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8","Type":"ContainerDied","Data":"bbd7f1a578fa0e066e8bffc51436d93488b6edb8dff9c9492a152b98ddc85666"}
Oct 29 23:15:19.561694 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.561436    5428 scope.go:122] "RemoveContainer" containerID="ba3d038c6a7ae5595eafc7e75580238245c17fcf6c8d401b14936647278a0403"
Oct 29 23:15:19.561927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.561908    5428 scope.go:122] "RemoveContainer" containerID="bbd7f1a578fa0e066e8bffc51436d93488b6edb8dff9c9492a152b98ddc85666"
Oct 29 23:15:19.562318 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:15:19.562293    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv_odh-ai-gateway-infra(7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv" podUID="7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8"
Oct 29 23:15:19.563227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.563206    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw"
Oct 29 23:15:19.563335 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.563214    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw" event={"ID":"536f4758-d66b-4908-96df-222a7fc64517","Type":"ContainerDied","Data":"53806aed65c74df6298637daf07e296208ed0c61894c6dddc5ee291e02fbd444"}
Oct 29 23:15:19.579994 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.579966    5428 scope.go:122] "RemoveContainer" containerID="26540de541a0de2c3f6056e9ae85a94086d31d5c41233366d0c426208dde361c"
Oct 29 23:15:19.652362 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.650248    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv"]
Oct 29 23:15:19.664341 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.663644    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw"]
Oct 29 23:15:19.669389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.669363    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7a3c1a-29888595-prwzw"]
Oct 29 23:15:19.750777 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.750752    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9"
Oct 29 23:15:19.758719 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.758694    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"]
Oct 29 23:15:19.759473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759458    5428 cpu_manager.go:414] "containerMap: removing container" podUID="bd15453b-4398-4e27-a18b-0b44251a8aff" containerName="cleanup"
Oct 29 23:15:19.759473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759473    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd15453b-4398-4e27-a18b-0b44251a8aff" containerName="cleanup"
Oct 29 23:15:19.759576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759484    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e9b5c2a2-7bce-4066-9390-7d2fac2998ab" containerName="cleanup"
Oct 29 23:15:19.759576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759489    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9b5c2a2-7bce-4066-9390-7d2fac2998ab" containerName="cleanup"
Oct 29 23:15:19.759576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759499    5428 cpu_manager.go:414] "containerMap: removing container" podUID="536f4758-d66b-4908-96df-222a7fc64517" containerName="cleanup"
Oct 29 23:15:19.759576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759505    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536f4758-d66b-4908-96df-222a7fc64517" containerName="cleanup"
Oct 29 23:15:19.759576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759534    5428 cpu_manager.go:414] "containerMap: removing container" podUID="536f4758-d66b-4908-96df-222a7fc64517" containerName="cleanup"
Oct 29 23:15:19.759576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759540    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536f4758-d66b-4908-96df-222a7fc64517" containerName="cleanup"
Oct 29 23:15:19.759576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759551    5428 cpu_manager.go:414] "containerMap: removing container" podUID="bd15453b-4398-4e27-a18b-0b44251a8aff" containerName="cleanup"
Oct 29 23:15:19.759576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759558    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd15453b-4398-4e27-a18b-0b44251a8aff" containerName="cleanup"
Oct 29 23:15:19.760014 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759589    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e9b5c2a2-7bce-4066-9390-7d2fac2998ab" containerName="cleanup"
Oct 29 23:15:19.760014 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759599    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9b5c2a2-7bce-4066-9390-7d2fac2998ab" containerName="cleanup"
Oct 29 23:15:19.760014 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759736    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="536f4758-d66b-4908-96df-222a7fc64517" containerName="cleanup"
Oct 29 23:15:19.760014 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759745    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="536f4758-d66b-4908-96df-222a7fc64517" containerName="cleanup"
Oct 29 23:15:19.760014 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759751    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd15453b-4398-4e27-a18b-0b44251a8aff" containerName="cleanup"
Oct 29 23:15:19.760014 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759757    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9b5c2a2-7bce-4066-9390-7d2fac2998ab" containerName="cleanup"
Oct 29 23:15:19.760014 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759765    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9b5c2a2-7bce-4066-9390-7d2fac2998ab" containerName="cleanup"
Oct 29 23:15:19.760014 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759773    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd15453b-4398-4e27-a18b-0b44251a8aff" containerName="cleanup"
Oct 29 23:15:19.760014 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.759779    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="536f4758-d66b-4908-96df-222a7fc64517" containerName="cleanup"
Oct 29 23:15:19.763679 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.763660    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"
Oct 29 23:15:19.788744 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.788312    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-2591f6-serving-cert\""
Oct 29 23:15:19.797852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.797818    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"]
Oct 29 23:15:19.844254 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.844157    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd15453b-4398-4e27-a18b-0b44251a8aff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd15453b-4398-4e27-a18b-0b44251a8aff-maas-api-service-ca\") pod \"bd15453b-4398-4e27-a18b-0b44251a8aff\" (UID: \"bd15453b-4398-4e27-a18b-0b44251a8aff\") "
Oct 29 23:15:19.844433 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.844337    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd15453b-4398-4e27-a18b-0b44251a8aff-kube-api-access-mcwmf\" (UniqueName: \"kubernetes.io/projected/bd15453b-4398-4e27-a18b-0b44251a8aff-kube-api-access-mcwmf\") pod \"bd15453b-4398-4e27-a18b-0b44251a8aff\" (UID: \"bd15453b-4398-4e27-a18b-0b44251a8aff\") "
Oct 29 23:15:19.844592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.844556    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afe8000b-075b-4052-b38c-fe2a222b8306-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s\" (UID: \"afe8000b-075b-4052-b38c-fe2a222b8306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"
Oct 29 23:15:19.844664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.844587    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd15453b-4398-4e27-a18b-0b44251a8aff-maas-api-service-ca" pod "bd15453b-4398-4e27-a18b-0b44251a8aff" (UID: "bd15453b-4398-4e27-a18b-0b44251a8aff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:19.844738 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.844720    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afe8000b-075b-4052-b38c-fe2a222b8306-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s\" (UID: \"afe8000b-075b-4052-b38c-fe2a222b8306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"
Oct 29 23:15:19.844894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.844875    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spw7l\" (UniqueName: \"kubernetes.io/projected/afe8000b-075b-4052-b38c-fe2a222b8306-kube-api-access-spw7l\") pod \"maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s\" (UID: \"afe8000b-075b-4052-b38c-fe2a222b8306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"
Oct 29 23:15:19.845027 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.844935    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afe8000b-075b-4052-b38c-fe2a222b8306-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s\" (UID: \"afe8000b-075b-4052-b38c-fe2a222b8306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"
Oct 29 23:15:19.845324 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.845276    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd15453b-4398-4e27-a18b-0b44251a8aff-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:15:19.847774 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.847744    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd15453b-4398-4e27-a18b-0b44251a8aff-kube-api-access-mcwmf" pod "bd15453b-4398-4e27-a18b-0b44251a8aff" (UID: "bd15453b-4398-4e27-a18b-0b44251a8aff"). InnerVolumeSpecName "kube-api-access-mcwmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:19.868628 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.868590    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:19.947348 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.947308    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afe8000b-075b-4052-b38c-fe2a222b8306-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s\" (UID: \"afe8000b-075b-4052-b38c-fe2a222b8306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"
Oct 29 23:15:19.947543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.947450    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afe8000b-075b-4052-b38c-fe2a222b8306-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s\" (UID: \"afe8000b-075b-4052-b38c-fe2a222b8306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"
Oct 29 23:15:19.947611 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.947541    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spw7l\" (UniqueName: \"kubernetes.io/projected/afe8000b-075b-4052-b38c-fe2a222b8306-kube-api-access-spw7l\") pod \"maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s\" (UID: \"afe8000b-075b-4052-b38c-fe2a222b8306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"
Oct 29 23:15:19.947611 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.947594    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afe8000b-075b-4052-b38c-fe2a222b8306-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s\" (UID: \"afe8000b-075b-4052-b38c-fe2a222b8306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"
Oct 29 23:15:19.948343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.948291    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcwmf\" (UniqueName: \"kubernetes.io/projected/bd15453b-4398-4e27-a18b-0b44251a8aff-kube-api-access-mcwmf\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:15:19.949034 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.949003    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afe8000b-075b-4052-b38c-fe2a222b8306-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s\" (UID: \"afe8000b-075b-4052-b38c-fe2a222b8306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"
Oct 29 23:15:19.950837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.950540    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afe8000b-075b-4052-b38c-fe2a222b8306-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s\" (UID: \"afe8000b-075b-4052-b38c-fe2a222b8306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"
Oct 29 23:15:19.961261 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.961239    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afe8000b-075b-4052-b38c-fe2a222b8306-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s\" (UID: \"afe8000b-075b-4052-b38c-fe2a222b8306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"
Oct 29 23:15:19.972033 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:19.972012    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spw7l\" (UniqueName: \"kubernetes.io/projected/afe8000b-075b-4052-b38c-fe2a222b8306-kube-api-access-spw7l\") pod \"maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s\" (UID: \"afe8000b-075b-4052-b38c-fe2a222b8306\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"
Oct 29 23:15:20.037061 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:20.037023    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:20.088709 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:20.088674    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"
Oct 29 23:15:20.205177 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:20.204479    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:20.417662 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:20.417635    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"]
Oct 29 23:15:20.418715 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:15:20.418687    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe8000b_075b_4052_b38c_fe2a222b8306.slice/crio-4e1a6c612fe57368a34e007d2d44bc39fcb5e0e752343254e9007f5066605c8c WatchSource:0}: Error finding container 4e1a6c612fe57368a34e007d2d44bc39fcb5e0e752343254e9007f5066605c8c: Status 404 returned error can't find the container with id 4e1a6c612fe57368a34e007d2d44bc39fcb5e0e752343254e9007f5066605c8c
Oct 29 23:15:20.577146 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:20.577116    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9"
Oct 29 23:15:20.577146 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:20.577128    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9" event={"ID":"bd15453b-4398-4e27-a18b-0b44251a8aff","Type":"ContainerDied","Data":"5a0ad3a25edf3b268cdbabc3dd5cde9c57ed288e87125291029318ef13cfdf95"}
Oct 29 23:15:20.577632 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:20.577199    5428 scope.go:122] "RemoveContainer" containerID="fa1607b8b9e31a796af33a5eb2223bdd875a5a24e533b07d07311c06770a5d7b"
Oct 29 23:15:20.578545 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:20.578525    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s" event={"ID":"afe8000b-075b-4052-b38c-fe2a222b8306","Type":"ContainerStarted","Data":"4e1a6c612fe57368a34e007d2d44bc39fcb5e0e752343254e9007f5066605c8c"}
Oct 29 23:15:20.680982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:20.676695    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9"]
Oct 29 23:15:20.680982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:20.680539    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1ee571-29888595-x8rh9"]
Oct 29 23:15:20.813493 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:20.813457    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="536f4758-d66b-4908-96df-222a7fc64517" path="/var/lib/kubelet/pods/536f4758-d66b-4908-96df-222a7fc64517/volumes"
Oct 29 23:15:20.814281 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:20.814245    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd15453b-4398-4e27-a18b-0b44251a8aff" path="/var/lib/kubelet/pods/bd15453b-4398-4e27-a18b-0b44251a8aff/volumes"
Oct 29 23:15:20.891174 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:20.891146    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv"
Oct 29 23:15:21.074824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:21.074769    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8-maas-api-service-ca\") pod \"7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8\" (UID: \"7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8\") "
Oct 29 23:15:21.074996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:21.074931    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8-kube-api-access-6r9nw\" (UniqueName: \"kubernetes.io/projected/7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8-kube-api-access-6r9nw\") pod \"7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8\" (UID: \"7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8\") "
Oct 29 23:15:21.075125 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:21.075103    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8-maas-api-service-ca" pod "7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8" (UID: "7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:21.075508 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:21.075492    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.077150 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:21.077125    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8-kube-api-access-6r9nw" pod "7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8" (UID: "7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8"). InnerVolumeSpecName "kube-api-access-6r9nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:21.213874 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:21.211858    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r9nw\" (UniqueName: \"kubernetes.io/projected/7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8-kube-api-access-6r9nw\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.593948 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:21.593844    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s" event={"ID":"afe8000b-075b-4052-b38c-fe2a222b8306","Type":"ContainerStarted","Data":"ea224f3cecd5286738db59b1a2496130dcc2fd05b7762a7166615ca160dc373e"}
Oct 29 23:15:21.593948 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:21.593921    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"
Oct 29 23:15:21.596332 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:21.596306    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv"
Oct 29 23:15:21.596445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:21.596329    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv" event={"ID":"7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8","Type":"ContainerDied","Data":"267a6e73abb43712e243a462f402afaf3540630da9fcd01700a844a485f6f4ca"}
Oct 29 23:15:21.596445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:21.596404    5428 scope.go:122] "RemoveContainer" containerID="bbd7f1a578fa0e066e8bffc51436d93488b6edb8dff9c9492a152b98ddc85666"
Oct 29 23:15:21.651422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:21.651379    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s" podStartSLOduration=2.2384100670000002 podStartE2EDuration="2.651365895s" totalImagesPullingTime="412.955828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:21.63663487 +0000 UTC m=+1924.979583618" watchObservedRunningTime="2026-10-29 23:15:21.651365895 +0000 UTC m=+1924.994314640"
Oct 29 23:15:21.678978 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:21.678949    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv"]
Oct 29 23:15:21.688851 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:21.688824    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-c47a72-29888595-l4znv"]
Oct 29 23:15:21.699165 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:21.699121    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:21.994518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:21.994482    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:22.108912 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:22.108876    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:22.230936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:22.230878    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:22.783148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:22.783110    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8" path="/var/lib/kubelet/pods/7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8/volumes"
Oct 29 23:15:23.776105 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:23.776018    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:24.203196 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:24.203159    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:25.593863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:25.593830    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:27.481444 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:27.481410    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:27.612708 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:27.612677    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"
Oct 29 23:15:28.196358 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:28.196322    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:28.735647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:28.735614    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:29.327016 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:29.326981    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:29.439400 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:29.439369    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:29.708091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:29.708044    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:31.674840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:31.674774    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:33.565528 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:33.565497    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:33.886458 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:33.886405    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:35.498619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:35.498585    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:37.093901 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:37.093863    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:38.800990 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:38.800956    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:41.127962 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:41.127926    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:43.282435 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:43.282401    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:44.933718 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:44.933690    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:46.640762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:46.640729    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:46.764972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:46.764940    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:47.371507 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:15:47.371471    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:15:48.762153 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:48.762113    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:50.592238 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.592207    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:50.671292 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.671257    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"]
Oct 29 23:15:50.673871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.673848    5428 cpu_manager.go:414] "containerMap: removing container" podUID="7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8" containerName="cleanup"
Oct 29 23:15:50.674062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.674047    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8" containerName="cleanup"
Oct 29 23:15:50.674158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.674146    5428 cpu_manager.go:414] "containerMap: removing container" podUID="7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8" containerName="cleanup"
Oct 29 23:15:50.674240 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.674230    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8" containerName="cleanup"
Oct 29 23:15:50.674359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.674344    5428 cpu_manager.go:414] "containerMap: removing container" podUID="7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8" containerName="cleanup"
Oct 29 23:15:50.674477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.674463    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8" containerName="cleanup"
Oct 29 23:15:50.674582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.674569    5428 cpu_manager.go:414] "containerMap: removing container" podUID="536f4758-d66b-4908-96df-222a7fc64517" containerName="cleanup"
Oct 29 23:15:50.674894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.674870    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536f4758-d66b-4908-96df-222a7fc64517" containerName="cleanup"
Oct 29 23:15:50.675013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.675001    5428 cpu_manager.go:414] "containerMap: removing container" podUID="bd15453b-4398-4e27-a18b-0b44251a8aff" containerName="cleanup"
Oct 29 23:15:50.675097 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.675086    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd15453b-4398-4e27-a18b-0b44251a8aff" containerName="cleanup"
Oct 29 23:15:50.675249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.675239    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e9b5c2a2-7bce-4066-9390-7d2fac2998ab" containerName="cleanup"
Oct 29 23:15:50.675327 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.675317    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9b5c2a2-7bce-4066-9390-7d2fac2998ab" containerName="cleanup"
Oct 29 23:15:50.675612 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.675593    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd15453b-4398-4e27-a18b-0b44251a8aff" containerName="cleanup"
Oct 29 23:15:50.675751 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.675739    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8" containerName="cleanup"
Oct 29 23:15:50.675853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.675842    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8" containerName="cleanup"
Oct 29 23:15:50.675942 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.675931    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c90c88f-d7b3-44f7-9ebb-ea9226ba92f8" containerName="cleanup"
Oct 29 23:15:50.676039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.676028    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9b5c2a2-7bce-4066-9390-7d2fac2998ab" containerName="cleanup"
Oct 29 23:15:50.686891 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.686868    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.690515 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.690488    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"]
Oct 29 23:15:50.702839 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.702276    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-2591f6-openshift-default-dockercfg-5rxmx\""
Oct 29 23:15:50.807047 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.807008    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-workload-socket\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.807226 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.807073    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-data\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.807226 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.807191    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-token\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.807354 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.807331    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-credential-socket\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.807425 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.807413    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istiod-ca-cert\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.807600 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.807580    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-podinfo\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.807684 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.807673    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-workload-certs\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.807725 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.807706    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-envoy\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.807791 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.807778    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9427\" (UniqueName: \"kubernetes.io/projected/72f44773-1ecf-4102-b29b-0b10e5cebaf1-kube-api-access-p9427\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.908949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.908916    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-credential-socket\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.909586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.909018    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istiod-ca-cert\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.909586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.909137    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-podinfo\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.909586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.909214    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-workload-certs\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.909586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.909239    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-envoy\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.909586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.909348    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-credential-socket\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.909586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.909363    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9427\" (UniqueName: \"kubernetes.io/projected/72f44773-1ecf-4102-b29b-0b10e5cebaf1-kube-api-access-p9427\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.909586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.909424    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-workload-socket\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.909586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.909489    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-data\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.909586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.909539    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-token\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.910233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.909730    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istiod-ca-cert\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.910233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.910034    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-workload-socket\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.910405 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.910381    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-workload-certs\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.910607 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.910471    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-data\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.913451 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.913426    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-envoy\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.914386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.914361    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-podinfo\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.935426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.935222    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9427\" (UniqueName: \"kubernetes.io/projected/72f44773-1ecf-4102-b29b-0b10e5cebaf1-kube-api-access-p9427\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.940345 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.940316    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-token\") pod \"e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") " pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:50.991231 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:50.991170    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:51.007364 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:51.007314    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:51.346567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:51.346538    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"]
Oct 29 23:15:51.349190 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:15:51.349160    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f44773_1ecf_4102_b29b_0b10e5cebaf1.slice/crio-6867206360ca438357e368774765d8a6ba313b61dbb74e6c8f79d55db2530266 WatchSource:0}: Error finding container 6867206360ca438357e368774765d8a6ba313b61dbb74e6c8f79d55db2530266: Status 404 returned error can't find the container with id 6867206360ca438357e368774765d8a6ba313b61dbb74e6c8f79d55db2530266
Oct 29 23:15:51.352676 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:51.352629    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:15:51.352784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:51.352713    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:15:51.352784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:51.352753    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:15:51.880550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:51.880514    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt" event={"ID":"72f44773-1ecf-4102-b29b-0b10e5cebaf1","Type":"ContainerStarted","Data":"2bfd13ece1de9e04d1dd1153cfe05a1d4d7a80730e2c08946899fab2d8e5355a"}
Oct 29 23:15:51.880989 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:51.880562    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt" event={"ID":"72f44773-1ecf-4102-b29b-0b10e5cebaf1","Type":"ContainerStarted","Data":"6867206360ca438357e368774765d8a6ba313b61dbb74e6c8f79d55db2530266"}
Oct 29 23:15:51.953019 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:51.952957    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt" podStartSLOduration=1.952940095 podStartE2EDuration="1.952940095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:51.936467287 +0000 UTC m=+1955.279416032" watchObservedRunningTime="2026-10-29 23:15:51.952940095 +0000 UTC m=+1955.295888841"
Oct 29 23:15:52.008306 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:52.008268    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:52.010545 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:52.010517    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 29 23:15:52.010706 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:52.010563    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt" podUID="72f44773-1ecf-4102-b29b-0b10e5cebaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 29 23:15:52.509377 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:52.508732    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:53.008435 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:53.008393    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 29 23:15:53.008887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:53.008453    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt" podUID="72f44773-1ecf-4102-b29b-0b10e5cebaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 29 23:15:54.008101 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:54.008070    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 29 23:15:54.008276 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:54.008120    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt" podUID="72f44773-1ecf-4102-b29b-0b10e5cebaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 29 23:15:54.196981 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:54.196940    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:55.007616 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:55.007584    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 29 23:15:55.007855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:55.007640    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt" podUID="72f44773-1ecf-4102-b29b-0b10e5cebaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 29 23:15:56.008133 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:56.008097    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 29 23:15:56.008526 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:56.008152    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt" podUID="72f44773-1ecf-4102-b29b-0b10e5cebaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 29 23:15:56.037795 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:56.037762    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:57.008244 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:57.008204    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 29 23:15:57.008727 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:57.008264    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt" podUID="72f44773-1ecf-4102-b29b-0b10e5cebaf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 29 23:15:57.388251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:57.388211    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf"]
Oct 29 23:15:57.406465 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:57.406436    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m"]
Oct 29 23:15:57.526542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:57.526504    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:57.648394 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:57.648270    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:58.071787 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:58.071699    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:58.077560 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:58.073352    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:58.077984 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:58.077964    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:15:58.251286 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:15:58.251240    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:00.283952 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:00.283910    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:01.215707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:01.215663    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:01.672743 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:01.672715    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:02.028458 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:02.028386    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:02.188008 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:02.187969    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:02.317652 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:02.317559    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:02.391824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:02.390864    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf" podUID="79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd" containerName="payload-processing" containerID="cri-o://a8bd3479d027cdf0e3bfb4aa6266e37e6d1cf480c143acc5b71dc6c61dc769c4" gracePeriod=25
Oct 29 23:16:02.427949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:02.427874    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m" podUID="1b715bb1-d85c-4b85-87d0-8a67112e3d4f" containerName="payload-pre-processing" containerID="cri-o://a0dc2fef0f3e39b7a06309f5141f505ed4e3532c848a1e0ed482f84ed3430272" gracePeriod=25
Oct 29 23:16:02.801609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:02.801582    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf"
Oct 29 23:16:02.846503 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:02.846359    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd-plugins-config-volume\") pod \"79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd\" (UID: \"79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd\") "
Oct 29 23:16:02.848263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:02.846657    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd-kube-api-access-zn6n9\" (UniqueName: \"kubernetes.io/projected/79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd-kube-api-access-zn6n9\") pod \"79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd\" (UID: \"79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd\") "
Oct 29 23:16:02.848263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:02.846881    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd-plugins-config-volume" pod "79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd" (UID: "79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:16:02.848263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:02.847521    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:16:02.850300 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:02.850277    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd-kube-api-access-zn6n9" pod "79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd" (UID: "79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd"). InnerVolumeSpecName "kube-api-access-zn6n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:16:02.856195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:02.856172    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m"
Oct 29 23:16:02.948243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:02.948198    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b715bb1-d85c-4b85-87d0-8a67112e3d4f-kube-api-access-pzz57\" (UniqueName: \"kubernetes.io/projected/1b715bb1-d85c-4b85-87d0-8a67112e3d4f-kube-api-access-pzz57\") pod \"1b715bb1-d85c-4b85-87d0-8a67112e3d4f\" (UID: \"1b715bb1-d85c-4b85-87d0-8a67112e3d4f\") "
Oct 29 23:16:02.948436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:02.948412    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b715bb1-d85c-4b85-87d0-8a67112e3d4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b715bb1-d85c-4b85-87d0-8a67112e3d4f-plugins-config-volume\") pod \"1b715bb1-d85c-4b85-87d0-8a67112e3d4f\" (UID: \"1b715bb1-d85c-4b85-87d0-8a67112e3d4f\") "
Oct 29 23:16:02.949311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:02.949254    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b715bb1-d85c-4b85-87d0-8a67112e3d4f-plugins-config-volume" pod "1b715bb1-d85c-4b85-87d0-8a67112e3d4f" (UID: "1b715bb1-d85c-4b85-87d0-8a67112e3d4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:16:02.949311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:02.949299    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn6n9\" (UniqueName: \"kubernetes.io/projected/79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd-kube-api-access-zn6n9\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:16:02.950850 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:02.950792    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b715bb1-d85c-4b85-87d0-8a67112e3d4f-kube-api-access-pzz57" pod "1b715bb1-d85c-4b85-87d0-8a67112e3d4f" (UID: "1b715bb1-d85c-4b85-87d0-8a67112e3d4f"). InnerVolumeSpecName "kube-api-access-pzz57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:16:02.976717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:02.976686    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:03.026146 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.026112    5428 generic.go:356] "Generic (PLEG): container finished" podID="79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd" containerID="a8bd3479d027cdf0e3bfb4aa6266e37e6d1cf480c143acc5b71dc6c61dc769c4" exitCode=0
Oct 29 23:16:03.026321 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.026155    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf"
Oct 29 23:16:03.026321 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.026201    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf" event={"ID":"79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd","Type":"ContainerDied","Data":"a8bd3479d027cdf0e3bfb4aa6266e37e6d1cf480c143acc5b71dc6c61dc769c4"}
Oct 29 23:16:03.026321 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.026255    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf" event={"ID":"79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd","Type":"ContainerDied","Data":"0bf8088d941e2f5bf9c47724070bc926f25e107cddf7d47b4907df0a0a9abbfd"}
Oct 29 23:16:03.026321 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.026292    5428 scope.go:122] "RemoveContainer" containerID="a8bd3479d027cdf0e3bfb4aa6266e37e6d1cf480c143acc5b71dc6c61dc769c4"
Oct 29 23:16:03.028001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.027967    5428 generic.go:356] "Generic (PLEG): container finished" podID="1b715bb1-d85c-4b85-87d0-8a67112e3d4f" containerID="a0dc2fef0f3e39b7a06309f5141f505ed4e3532c848a1e0ed482f84ed3430272" exitCode=0
Oct 29 23:16:03.028113 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.028023    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m"
Oct 29 23:16:03.028191 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.028163    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m" event={"ID":"1b715bb1-d85c-4b85-87d0-8a67112e3d4f","Type":"ContainerDied","Data":"a0dc2fef0f3e39b7a06309f5141f505ed4e3532c848a1e0ed482f84ed3430272"}
Oct 29 23:16:03.028269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.028239    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m" event={"ID":"1b715bb1-d85c-4b85-87d0-8a67112e3d4f","Type":"ContainerDied","Data":"a7a01578c5917cf2e9995e0c06692f2c3f200661186d944ee2936f411d865738"}
Oct 29 23:16:03.039711 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.039665    5428 scope.go:122] "RemoveContainer" containerID="a8bd3479d027cdf0e3bfb4aa6266e37e6d1cf480c143acc5b71dc6c61dc769c4"
Oct 29 23:16:03.039989 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:16:03.039969    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8bd3479d027cdf0e3bfb4aa6266e37e6d1cf480c143acc5b71dc6c61dc769c4\": container with ID starting with a8bd3479d027cdf0e3bfb4aa6266e37e6d1cf480c143acc5b71dc6c61dc769c4 not found: ID does not exist" containerID="a8bd3479d027cdf0e3bfb4aa6266e37e6d1cf480c143acc5b71dc6c61dc769c4"
Oct 29 23:16:03.040044 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.039996    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8bd3479d027cdf0e3bfb4aa6266e37e6d1cf480c143acc5b71dc6c61dc769c4"} err="failed to get container status \"a8bd3479d027cdf0e3bfb4aa6266e37e6d1cf480c143acc5b71dc6c61dc769c4\": rpc error: code = NotFound desc = could not find container \"a8bd3479d027cdf0e3bfb4aa6266e37e6d1cf480c143acc5b71dc6c61dc769c4\": container with ID starting with a8bd3479d027cdf0e3bfb4aa6266e37e6d1cf480c143acc5b71dc6c61dc769c4 not found: ID does not exist"
Oct 29 23:16:03.040044 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.040015    5428 scope.go:122] "RemoveContainer" containerID="a0dc2fef0f3e39b7a06309f5141f505ed4e3532c848a1e0ed482f84ed3430272"
Oct 29 23:16:03.050298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.050266    5428 scope.go:122] "RemoveContainer" containerID="a0dc2fef0f3e39b7a06309f5141f505ed4e3532c848a1e0ed482f84ed3430272"
Oct 29 23:16:03.050583 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:16:03.050562    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0dc2fef0f3e39b7a06309f5141f505ed4e3532c848a1e0ed482f84ed3430272\": container with ID starting with a0dc2fef0f3e39b7a06309f5141f505ed4e3532c848a1e0ed482f84ed3430272 not found: ID does not exist" containerID="a0dc2fef0f3e39b7a06309f5141f505ed4e3532c848a1e0ed482f84ed3430272"
Oct 29 23:16:03.050642 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.050595    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0dc2fef0f3e39b7a06309f5141f505ed4e3532c848a1e0ed482f84ed3430272"} err="failed to get container status \"a0dc2fef0f3e39b7a06309f5141f505ed4e3532c848a1e0ed482f84ed3430272\": rpc error: code = NotFound desc = could not find container \"a0dc2fef0f3e39b7a06309f5141f505ed4e3532c848a1e0ed482f84ed3430272\": container with ID starting with a0dc2fef0f3e39b7a06309f5141f505ed4e3532c848a1e0ed482f84ed3430272 not found: ID does not exist"
Oct 29 23:16:03.051031 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.051011    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b715bb1-d85c-4b85-87d0-8a67112e3d4f-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:16:03.051096 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.051033    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzz57\" (UniqueName: \"kubernetes.io/projected/1b715bb1-d85c-4b85-87d0-8a67112e3d4f-kube-api-access-pzz57\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:16:03.080332 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.080300    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m"]
Oct 29 23:16:03.091271 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.091243    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c47a72-6cbdc45bd7-pvk8m"]
Oct 29 23:16:03.120780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.120747    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf"]
Oct 29 23:16:03.127097 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.127074    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c47a72-84f98868f4-9nzdf"]
Oct 29 23:16:03.139570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.139549    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:03.283118 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.283088    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:03.661749 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:03.661715    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:04.785780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:04.785749    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b715bb1-d85c-4b85-87d0-8a67112e3d4f" path="/var/lib/kubelet/pods/1b715bb1-d85c-4b85-87d0-8a67112e3d4f/volumes"
Oct 29 23:16:04.786455 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:04.786435    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd" path="/var/lib/kubelet/pods/79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd/volumes"
Oct 29 23:16:05.149835 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:05.149785    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:05.278721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:05.278667    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:06.156217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:06.156183    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:06.279677 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:06.279641    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:07.198774 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:07.198736    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:07.851842 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:07.851793    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:07.968116 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:07.968083    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:08.026307 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.026266    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn"]
Oct 29 23:16:08.027107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.027090    5428 cpu_manager.go:414] "containerMap: removing container" podUID="1b715bb1-d85c-4b85-87d0-8a67112e3d4f" containerName="payload-pre-processing"
Oct 29 23:16:08.027179 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.027109    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b715bb1-d85c-4b85-87d0-8a67112e3d4f" containerName="payload-pre-processing"
Oct 29 23:16:08.027179 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.027149    5428 cpu_manager.go:414] "containerMap: removing container" podUID="79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd" containerName="payload-processing"
Oct 29 23:16:08.027179 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.027154    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd" containerName="payload-processing"
Oct 29 23:16:08.027284 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.027260    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b715bb1-d85c-4b85-87d0-8a67112e3d4f" containerName="payload-pre-processing"
Oct 29 23:16:08.027284 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.027276    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d1fa8d-832d-4a58-b7d6-b4cbd7e06cfd" containerName="payload-processing"
Oct 29 23:16:08.032235 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.032216    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn"
Oct 29 23:16:08.071357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.071328    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-2b0108-dockercfg-ct59l\""
Oct 29 23:16:08.071626 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.071602    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-2b0108\""
Oct 29 23:16:08.086688 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.086652    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn"]
Oct 29 23:16:08.168706 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.168668    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx4kj\" (UniqueName: \"kubernetes.io/projected/8755d1b0-bf1b-45c5-8923-b2e448d967f7-kube-api-access-hx4kj\") pod \"payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn\" (UID: \"8755d1b0-bf1b-45c5-8923-b2e448d967f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn"
Oct 29 23:16:08.192676 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.192642    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8755d1b0-bf1b-45c5-8923-b2e448d967f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn\" (UID: \"8755d1b0-bf1b-45c5-8923-b2e448d967f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn"
Oct 29 23:16:08.342246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.342207    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8755d1b0-bf1b-45c5-8923-b2e448d967f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn\" (UID: \"8755d1b0-bf1b-45c5-8923-b2e448d967f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn"
Oct 29 23:16:08.342792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.342397    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx4kj\" (UniqueName: \"kubernetes.io/projected/8755d1b0-bf1b-45c5-8923-b2e448d967f7-kube-api-access-hx4kj\") pod \"payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn\" (UID: \"8755d1b0-bf1b-45c5-8923-b2e448d967f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn"
Oct 29 23:16:08.343684 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.343655    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8755d1b0-bf1b-45c5-8923-b2e448d967f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn\" (UID: \"8755d1b0-bf1b-45c5-8923-b2e448d967f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn"
Oct 29 23:16:08.366324 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.366297    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx4kj\" (UniqueName: \"kubernetes.io/projected/8755d1b0-bf1b-45c5-8923-b2e448d967f7-kube-api-access-hx4kj\") pod \"payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn\" (UID: \"8755d1b0-bf1b-45c5-8923-b2e448d967f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn"
Oct 29 23:16:08.526501 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.526396    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn"
Oct 29 23:16:08.851896 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:16:08.851866    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8755d1b0_bf1b_45c5_8923_b2e448d967f7.slice/crio-b149aa26c0bce174a32f53720b4b6e9af1ccea14c27a2418eeb8d3a9b780093b WatchSource:0}: Error finding container b149aa26c0bce174a32f53720b4b6e9af1ccea14c27a2418eeb8d3a9b780093b: Status 404 returned error can't find the container with id b149aa26c0bce174a32f53720b4b6e9af1ccea14c27a2418eeb8d3a9b780093b
Oct 29 23:16:08.852703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:08.852675    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn"]
Oct 29 23:16:09.100520 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:09.100413    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn" event={"ID":"8755d1b0-bf1b-45c5-8923-b2e448d967f7","Type":"ContainerStarted","Data":"fecffd2c39a1ebb251a915ef45ca30ff89b79f5c78ac4b286d7f82530b932f5b"}
Oct 29 23:16:09.100520 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:09.100503    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn" event={"ID":"8755d1b0-bf1b-45c5-8923-b2e448d967f7","Type":"ContainerStarted","Data":"b149aa26c0bce174a32f53720b4b6e9af1ccea14c27a2418eeb8d3a9b780093b"}
Oct 29 23:16:09.100735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:09.100649    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn"
Oct 29 23:16:09.145859 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:09.145790    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn" podStartSLOduration=2.145778384 podStartE2EDuration="2.145778384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:09.141053411 +0000 UTC m=+1972.484002156" watchObservedRunningTime="2026-10-29 23:16:09.145778384 +0000 UTC m=+1972.488727129"
Oct 29 23:16:09.505058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:09.505019    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:10.218087 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:10.218036    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:10.332403 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:10.332370    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:10.573901 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:10.573780    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:11.496272 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:11.496240    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:13.243125 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:13.243092    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:14.939887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:14.939851    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:16.415996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:16.415963    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:16.805631 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:16.805549    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:16.919864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:16.919792    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:17.063902 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:17.061647    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:17.373153 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:16:17.373116    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:16:18.696349 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:18.696314    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:20.110979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:20.110951    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn"
Oct 29 23:16:20.427204 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:20.427175    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:20.556837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:20.556743    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:20.706636 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:20.706536    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:22.133618 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:22.133584    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:22.282740 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:22.282702    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:22.307295 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:22.307262    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj"]
Oct 29 23:16:22.314097 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:22.314062    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj"
Oct 29 23:16:22.316949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:22.316919    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7txk\" (UniqueName: \"kubernetes.io/projected/999d9e5a-1f9f-4204-ba58-18317566426b-kube-api-access-r7txk\") pod \"payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj\" (UID: \"999d9e5a-1f9f-4204-ba58-18317566426b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj"
Oct 29 23:16:22.317129 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:22.317109    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999d9e5a-1f9f-4204-ba58-18317566426b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj\" (UID: \"999d9e5a-1f9f-4204-ba58-18317566426b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj"
Oct 29 23:16:22.332069 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:22.332047    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj"]
Oct 29 23:16:22.337543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:22.337497    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-a4f2db-dockercfg-b7lv4\""
Oct 29 23:16:22.383716 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:22.380681    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-a4f2db\""
Oct 29 23:16:22.446850 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:22.424011    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7txk\" (UniqueName: \"kubernetes.io/projected/999d9e5a-1f9f-4204-ba58-18317566426b-kube-api-access-r7txk\") pod \"payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj\" (UID: \"999d9e5a-1f9f-4204-ba58-18317566426b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj"
Oct 29 23:16:22.465897 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:22.465855    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999d9e5a-1f9f-4204-ba58-18317566426b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj\" (UID: \"999d9e5a-1f9f-4204-ba58-18317566426b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj"
Oct 29 23:16:22.483860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:22.475092    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999d9e5a-1f9f-4204-ba58-18317566426b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj\" (UID: \"999d9e5a-1f9f-4204-ba58-18317566426b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj"
Oct 29 23:16:22.508830 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:22.494621    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7txk\" (UniqueName: \"kubernetes.io/projected/999d9e5a-1f9f-4204-ba58-18317566426b-kube-api-access-r7txk\") pod \"payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj\" (UID: \"999d9e5a-1f9f-4204-ba58-18317566426b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj"
Oct 29 23:16:22.611870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:22.611837    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:22.643984 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:22.643911    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj"
Oct 29 23:16:22.983657 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:22.983624    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj"]
Oct 29 23:16:22.984131 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:16:22.984100    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999d9e5a_1f9f_4204_ba58_18317566426b.slice/crio-1f9f4f2f3b3ab6bda11770f36e94e069d0dabe14d6989f76246fd31e84f920a2 WatchSource:0}: Error finding container 1f9f4f2f3b3ab6bda11770f36e94e069d0dabe14d6989f76246fd31e84f920a2: Status 404 returned error can't find the container with id 1f9f4f2f3b3ab6bda11770f36e94e069d0dabe14d6989f76246fd31e84f920a2
Oct 29 23:16:23.220287 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:23.220255    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj" event={"ID":"999d9e5a-1f9f-4204-ba58-18317566426b","Type":"ContainerStarted","Data":"fd11fd86f1b9c68a355a186e76358a1e6444b0804611f9a3a3eb4bf98bc8a019"}
Oct 29 23:16:23.220664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:23.220304    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj" event={"ID":"999d9e5a-1f9f-4204-ba58-18317566426b","Type":"ContainerStarted","Data":"1f9f4f2f3b3ab6bda11770f36e94e069d0dabe14d6989f76246fd31e84f920a2"}
Oct 29 23:16:23.220664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:23.220486    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj"
Oct 29 23:16:23.274084 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:23.273994    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj" podStartSLOduration=1.273979836 podStartE2EDuration="1.273979836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:23.271235797 +0000 UTC m=+1986.614184542" watchObservedRunningTime="2026-10-29 23:16:23.273979836 +0000 UTC m=+1986.616928581"
Oct 29 23:16:24.551637 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:24.551603    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:26.250024 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:26.249991    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:27.785387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:27.785180    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:28.607582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:28.607508    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:28.751958 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:28.751917    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:30.803439 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:30.803319    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:32.684646 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:32.684608    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:33.359012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:33.358974    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:34.232774 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:34.232745    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj"
Oct 29 23:16:34.725879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:34.725844    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:36.489434 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:36.489392    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:38.281128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:38.281090    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:40.086344 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:40.086311    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:40.534481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:40.534450    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:40.668459 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:40.668422    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:41.734575 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:41.734538    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"]
Oct 29 23:16:41.739533 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:41.739507    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:16:41.760989 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:41.760958    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"]
Oct 29 23:16:41.766447 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:41.766422    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-2b0108-serving-cert\""
Oct 29 23:16:41.801449 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:41.801413    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wlvq\" (UniqueName: \"kubernetes.io/projected/cb6ed96b-0059-4994-a0cd-8543d915a176-kube-api-access-5wlvq\") pod \"maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq\" (UID: \"cb6ed96b-0059-4994-a0cd-8543d915a176\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:16:41.801636 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:41.801615    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6ed96b-0059-4994-a0cd-8543d915a176-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq\" (UID: \"cb6ed96b-0059-4994-a0cd-8543d915a176\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:16:41.801786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:41.801768    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6ed96b-0059-4994-a0cd-8543d915a176-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq\" (UID: \"cb6ed96b-0059-4994-a0cd-8543d915a176\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:16:41.801890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:41.801875    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6ed96b-0059-4994-a0cd-8543d915a176-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq\" (UID: \"cb6ed96b-0059-4994-a0cd-8543d915a176\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:16:41.832789 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:41.832762    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:41.903044 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:41.903014    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6ed96b-0059-4994-a0cd-8543d915a176-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq\" (UID: \"cb6ed96b-0059-4994-a0cd-8543d915a176\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:16:41.903210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:41.903070    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6ed96b-0059-4994-a0cd-8543d915a176-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq\" (UID: \"cb6ed96b-0059-4994-a0cd-8543d915a176\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:16:41.903210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:41.903102    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wlvq\" (UniqueName: \"kubernetes.io/projected/cb6ed96b-0059-4994-a0cd-8543d915a176-kube-api-access-5wlvq\") pod \"maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq\" (UID: \"cb6ed96b-0059-4994-a0cd-8543d915a176\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:16:41.903341 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:16:41.903203    5428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-serving-cert: secret "maas-api-e2e-shared-b-w6-2b0108-serving-cert" not found
Oct 29 23:16:41.903341 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:16:41.903283    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb6ed96b-0059-4994-a0cd-8543d915a176-maas-api-tls podName:cb6ed96b-0059-4994-a0cd-8543d915a176 nodeName:}" failed. No retries permitted until 2026-10-29 23:16:42.403263287 +0000 UTC m=+2005.746212027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb6ed96b-0059-4994-a0cd-8543d915a176-maas-api-tls") pod "maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq" (UID: "cb6ed96b-0059-4994-a0cd-8543d915a176") : secret "maas-api-e2e-shared-b-w6-2b0108-serving-cert" not found
Oct 29 23:16:41.903341 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:41.903205    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6ed96b-0059-4994-a0cd-8543d915a176-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq\" (UID: \"cb6ed96b-0059-4994-a0cd-8543d915a176\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:16:41.903889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:41.903872    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6ed96b-0059-4994-a0cd-8543d915a176-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq\" (UID: \"cb6ed96b-0059-4994-a0cd-8543d915a176\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:16:41.905452 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:41.905423    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6ed96b-0059-4994-a0cd-8543d915a176-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq\" (UID: \"cb6ed96b-0059-4994-a0cd-8543d915a176\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:16:41.947363 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:41.946555    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wlvq\" (UniqueName: \"kubernetes.io/projected/cb6ed96b-0059-4994-a0cd-8543d915a176-kube-api-access-5wlvq\") pod \"maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq\" (UID: \"cb6ed96b-0059-4994-a0cd-8543d915a176\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:16:41.953311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:41.953273    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:42.071785 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:42.071702    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:42.410671 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:42.410644    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6ed96b-0059-4994-a0cd-8543d915a176-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq\" (UID: \"cb6ed96b-0059-4994-a0cd-8543d915a176\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:16:42.413120 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:42.413098    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6ed96b-0059-4994-a0cd-8543d915a176-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq\" (UID: \"cb6ed96b-0059-4994-a0cd-8543d915a176\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:16:42.655602 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:42.655564    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:16:42.988912 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:42.988885    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"]
Oct 29 23:16:42.989993 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:16:42.989965    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6ed96b_0059_4994_a0cd_8543d915a176.slice/crio-b95746d0a24cf2955a8290e369049f75e00c1ff86cc7905e2b7e2ea887924502 WatchSource:0}: Error finding container b95746d0a24cf2955a8290e369049f75e00c1ff86cc7905e2b7e2ea887924502: Status 404 returned error can't find the container with id b95746d0a24cf2955a8290e369049f75e00c1ff86cc7905e2b7e2ea887924502
Oct 29 23:16:42.991566 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:42.991547    5428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:16:43.398358 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:43.398331    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq" event={"ID":"cb6ed96b-0059-4994-a0cd-8543d915a176","Type":"ContainerStarted","Data":"b95746d0a24cf2955a8290e369049f75e00c1ff86cc7905e2b7e2ea887924502"}
Oct 29 23:16:44.098425 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:44.098394    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:44.408880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:44.408842    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq" event={"ID":"cb6ed96b-0059-4994-a0cd-8543d915a176","Type":"ContainerStarted","Data":"88638fc1ad71c5f201e44ae1c7b613728dd9fe6673dc8f0f4e41cc192d122f35"}
Oct 29 23:16:44.409056 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:44.408912    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:16:44.461004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:44.460946    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq" podStartSLOduration=3.081593032 podStartE2EDuration="3.460927794s" totalImagesPullingTime="379.334762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:44.453145959 +0000 UTC m=+2007.796094705" watchObservedRunningTime="2026-10-29 23:16:44.460927794 +0000 UTC m=+2007.803876551"
Oct 29 23:16:44.855704 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:44.855613    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:45.891453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:45.891417    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:47.370079 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:16:47.370031    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:16:47.571125 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:47.571089    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:48.324267 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:48.324236    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:49.362765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:49.362727    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:50.422645 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:50.422614    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:16:51.062060 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:51.062023    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:52.803089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:52.803056    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:54.572541 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:54.572499    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:56.989603 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:56.989566    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:58.767785 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:58.767752    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"]
Oct 29 23:16:58.774302 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:58.774272    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"
Oct 29 23:16:58.792838 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:58.792771    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-a4f2db-serving-cert\""
Oct 29 23:16:58.796154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:58.796126    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"]
Oct 29 23:16:58.858480 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:58.858433    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:58.908347 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:58.908316    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa67df40-c3c5-4e47-b3cb-bc5780382c69-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq\" (UID: \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"
Oct 29 23:16:58.908532 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:58.908384    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa67df40-c3c5-4e47-b3cb-bc5780382c69-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq\" (UID: \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"
Oct 29 23:16:58.908588 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:58.908541    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bklq5\" (UniqueName: \"kubernetes.io/projected/fa67df40-c3c5-4e47-b3cb-bc5780382c69-kube-api-access-bklq5\") pod \"maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq\" (UID: \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"
Oct 29 23:16:58.908636 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:58.908584    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa67df40-c3c5-4e47-b3cb-bc5780382c69-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq\" (UID: \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"
Oct 29 23:16:58.976303 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:58.976261    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:59.010561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:59.010504    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa67df40-c3c5-4e47-b3cb-bc5780382c69-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq\" (UID: \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"
Oct 29 23:16:59.010776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:59.010610    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa67df40-c3c5-4e47-b3cb-bc5780382c69-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq\" (UID: \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"
Oct 29 23:16:59.010776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:59.010737    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bklq5\" (UniqueName: \"kubernetes.io/projected/fa67df40-c3c5-4e47-b3cb-bc5780382c69-kube-api-access-bklq5\") pod \"maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq\" (UID: \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"
Oct 29 23:16:59.010980 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:59.010777    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa67df40-c3c5-4e47-b3cb-bc5780382c69-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq\" (UID: \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"
Oct 29 23:16:59.012081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:59.012044    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa67df40-c3c5-4e47-b3cb-bc5780382c69-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq\" (UID: \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"
Oct 29 23:16:59.020077 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:59.019956    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa67df40-c3c5-4e47-b3cb-bc5780382c69-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq\" (UID: \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"
Oct 29 23:16:59.022027 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:59.021966    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa67df40-c3c5-4e47-b3cb-bc5780382c69-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq\" (UID: \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"
Oct 29 23:16:59.037514 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:59.037479    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bklq5\" (UniqueName: \"kubernetes.io/projected/fa67df40-c3c5-4e47-b3cb-bc5780382c69-kube-api-access-bklq5\") pod \"maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq\" (UID: \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"
Oct 29 23:16:59.095969 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:59.095927    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"
Oct 29 23:16:59.424835 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:59.424792    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"]
Oct 29 23:16:59.425981 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:16:59.425955    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa67df40_c3c5_4e47_b3cb_bc5780382c69.slice/crio-cf5fadae66f282580ad7380f88ec698792e1d5052a8a2d6b83110ac526654a4e WatchSource:0}: Error finding container cf5fadae66f282580ad7380f88ec698792e1d5052a8a2d6b83110ac526654a4e: Status 404 returned error can't find the container with id cf5fadae66f282580ad7380f88ec698792e1d5052a8a2d6b83110ac526654a4e
Oct 29 23:16:59.545163 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:16:59.545128    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq" event={"ID":"fa67df40-c3c5-4e47-b3cb-bc5780382c69","Type":"ContainerStarted","Data":"cf5fadae66f282580ad7380f88ec698792e1d5052a8a2d6b83110ac526654a4e"}
Oct 29 23:17:00.554751 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:00.554712    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq" event={"ID":"fa67df40-c3c5-4e47-b3cb-bc5780382c69","Type":"ContainerStarted","Data":"4fa9a55abf57a0bce709381b2b6695b6323ca9fc4643c8fe2a92e9eb67120b93"}
Oct 29 23:17:00.555238 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:00.554829    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"
Oct 29 23:17:00.596755 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:00.596703    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq" podStartSLOduration=2.262696393 podStartE2EDuration="2.596686687s" totalImagesPullingTime="333.990294ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:00.596233963 +0000 UTC m=+2023.939182706" watchObservedRunningTime="2026-10-29 23:17:00.596686687 +0000 UTC m=+2023.939635433"
Oct 29 23:17:00.752252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:00.752220    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:02.503870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:02.503838    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:04.267766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:04.267733    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:06.004387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:06.004343    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:06.569154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:06.569125    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"
Oct 29 23:17:07.614843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:07.614793    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:09.370060 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:09.370024    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:11.006479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:11.006450    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:12.773969 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:12.773937    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:14.550949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:14.550921    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:16.176052 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:16.176009    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:17.368494 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:17:17.368454    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:17:17.888756 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:17.888720    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:19.556307 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:19.556271    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:21.277678 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:21.277639    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:22.850603 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:22.850573    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:23.236858 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:23.236828    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:24.994599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:24.994566    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:25.203299 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:25.203267    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:25.708832 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:25.708765    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:26.056483 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:26.056392    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:26.983221 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:26.983187    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:28.687489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:28.687454    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:28.866552 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:28.866520    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:29.234091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:29.234062    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:29.824399 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:29.824371    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:29.969943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:29.969901    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:30.783945 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:30.783915    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:31.531464 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:31.531432    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:31.644940 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:31.644908    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:32.772227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:32.772195    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:34.583226 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:34.583193    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:36.261942 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:36.261908    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:38.180641 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:38.180597    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:39.931990 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:39.931959    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:41.627092 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:41.627045    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:42.154821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:42.150813    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:42.627285 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:42.627249    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:43.118786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:43.118751    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:43.651724 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:43.651529    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:44.677480 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:44.677441    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:46.702344 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:46.702307    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:47.369104 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:17:47.369070    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:17:48.713398 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:48.713368    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:49.209256 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:49.209219    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:49.363159 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:49.363122    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:50.859320 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:50.859275    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:51.015760 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.015730    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd"]
Oct 29 23:17:51.020849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.020833    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd"
Oct 29 23:17:51.028957 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.028751    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:51.039253 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.038952    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-383f0e\""
Oct 29 23:17:51.048367 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.048347    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-383f0e-dockercfg-l568s\""
Oct 29 23:17:51.055875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.055846    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd"]
Oct 29 23:17:51.185852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.185782    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45"]
Oct 29 23:17:51.192619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.192590    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45"
Oct 29 23:17:51.318903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.318867    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45"]
Oct 29 23:17:51.324424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.320222    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f34098a-acd0-4c75-b946-0de1616e0ba8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd\" (UID: \"1f34098a-acd0-4c75-b946-0de1616e0ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd"
Oct 29 23:17:51.324809 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.324761    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8q8v\" (UniqueName: \"kubernetes.io/projected/1f34098a-acd0-4c75-b946-0de1616e0ba8-kube-api-access-z8q8v\") pod \"payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd\" (UID: \"1f34098a-acd0-4c75-b946-0de1616e0ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd"
Oct 29 23:17:51.426187 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.426137    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b46913cc-66af-454a-8d7f-0cff4a4b9d76-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45\" (UID: \"b46913cc-66af-454a-8d7f-0cff4a4b9d76\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45"
Oct 29 23:17:51.426502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.426482    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f34098a-acd0-4c75-b946-0de1616e0ba8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd\" (UID: \"1f34098a-acd0-4c75-b946-0de1616e0ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd"
Oct 29 23:17:51.426568 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.426541    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8twr2\" (UniqueName: \"kubernetes.io/projected/b46913cc-66af-454a-8d7f-0cff4a4b9d76-kube-api-access-8twr2\") pod \"payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45\" (UID: \"b46913cc-66af-454a-8d7f-0cff4a4b9d76\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45"
Oct 29 23:17:51.426649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.426627    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8q8v\" (UniqueName: \"kubernetes.io/projected/1f34098a-acd0-4c75-b946-0de1616e0ba8-kube-api-access-z8q8v\") pod \"payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd\" (UID: \"1f34098a-acd0-4c75-b946-0de1616e0ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd"
Oct 29 23:17:51.427268 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.427242    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f34098a-acd0-4c75-b946-0de1616e0ba8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd\" (UID: \"1f34098a-acd0-4c75-b946-0de1616e0ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd"
Oct 29 23:17:51.450318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.450251    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8q8v\" (UniqueName: \"kubernetes.io/projected/1f34098a-acd0-4c75-b946-0de1616e0ba8-kube-api-access-z8q8v\") pod \"payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd\" (UID: \"1f34098a-acd0-4c75-b946-0de1616e0ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd"
Oct 29 23:17:51.528696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.528656    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8twr2\" (UniqueName: \"kubernetes.io/projected/b46913cc-66af-454a-8d7f-0cff4a4b9d76-kube-api-access-8twr2\") pod \"payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45\" (UID: \"b46913cc-66af-454a-8d7f-0cff4a4b9d76\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45"
Oct 29 23:17:51.528947 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.528796    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b46913cc-66af-454a-8d7f-0cff4a4b9d76-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45\" (UID: \"b46913cc-66af-454a-8d7f-0cff4a4b9d76\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45"
Oct 29 23:17:51.529715 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.529686    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b46913cc-66af-454a-8d7f-0cff4a4b9d76-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45\" (UID: \"b46913cc-66af-454a-8d7f-0cff4a4b9d76\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45"
Oct 29 23:17:51.552437 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.552409    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8twr2\" (UniqueName: \"kubernetes.io/projected/b46913cc-66af-454a-8d7f-0cff4a4b9d76-kube-api-access-8twr2\") pod \"payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45\" (UID: \"b46913cc-66af-454a-8d7f-0cff4a4b9d76\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45"
Oct 29 23:17:51.635544 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.635499    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd"
Oct 29 23:17:51.848816 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.848781    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45"
Oct 29 23:17:51.947769 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:51.947737    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd"]
Oct 29 23:17:51.949047 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:17:51.949009    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f34098a_acd0_4c75_b946_0de1616e0ba8.slice/crio-0e9190b92f117abef6879e17d0994eeb4b598cc9e255b3220874b00ece5d916c WatchSource:0}: Error finding container 0e9190b92f117abef6879e17d0994eeb4b598cc9e255b3220874b00ece5d916c: Status 404 returned error can't find the container with id 0e9190b92f117abef6879e17d0994eeb4b598cc9e255b3220874b00ece5d916c
Oct 29 23:17:52.160175 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:52.160144    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45"]
Oct 29 23:17:52.160707 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:17:52.160678    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46913cc_66af_454a_8d7f_0cff4a4b9d76.slice/crio-15b47c7adaf57036187178cef3da4d7d81c0b7115ac7e7672e360a678c31a4ad WatchSource:0}: Error finding container 15b47c7adaf57036187178cef3da4d7d81c0b7115ac7e7672e360a678c31a4ad: Status 404 returned error can't find the container with id 15b47c7adaf57036187178cef3da4d7d81c0b7115ac7e7672e360a678c31a4ad
Oct 29 23:17:52.298536 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:52.298502    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45" event={"ID":"b46913cc-66af-454a-8d7f-0cff4a4b9d76","Type":"ContainerStarted","Data":"ad0013336aa7bc0dfe7dcd8c8ddb4824b4182e255024216a5452470e21e90eae"}
Oct 29 23:17:52.298723 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:52.298563    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45" event={"ID":"b46913cc-66af-454a-8d7f-0cff4a4b9d76","Type":"ContainerStarted","Data":"15b47c7adaf57036187178cef3da4d7d81c0b7115ac7e7672e360a678c31a4ad"}
Oct 29 23:17:52.298723 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:52.298594    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45"
Oct 29 23:17:52.300191 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:52.300164    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd" event={"ID":"1f34098a-acd0-4c75-b946-0de1616e0ba8","Type":"ContainerStarted","Data":"40c015e19e10c402eb3e49c8601aa5817f2a1bf23f72cbe52a645d26331ab21b"}
Oct 29 23:17:52.300324 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:52.300214    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd"
Oct 29 23:17:52.300324 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:52.300236    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd" event={"ID":"1f34098a-acd0-4c75-b946-0de1616e0ba8","Type":"ContainerStarted","Data":"0e9190b92f117abef6879e17d0994eeb4b598cc9e255b3220874b00ece5d916c"}
Oct 29 23:17:52.345907 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:52.345854    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45" podStartSLOduration=1.345839148 podStartE2EDuration="1.345839148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:52.339673671 +0000 UTC m=+2075.682622435" watchObservedRunningTime="2026-10-29 23:17:52.345839148 +0000 UTC m=+2075.688787894"
Oct 29 23:17:52.383538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:52.383489    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd" podStartSLOduration=2.3834773 podStartE2EDuration="2.3834773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:52.380616115 +0000 UTC m=+2075.723564861" watchObservedRunningTime="2026-10-29 23:17:52.3834773 +0000 UTC m=+2075.726426087"
Oct 29 23:17:53.162357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:53.162316    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:54.934225 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:54.934194    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:56.640789 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:56.640751    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:58.397728 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:17:58.397644    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:00.252879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:00.252815    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:00.443916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:00.443853    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:00.563635 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:00.563540    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:02.205758 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:02.205694    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:03.315957 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:03.315928    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd"
Oct 29 23:18:03.316424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:03.316012    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45"
Oct 29 23:18:03.988891 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:03.988844    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:05.862403 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:05.862352    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:07.565961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:07.565928    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:09.138655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.138620    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"]
Oct 29 23:18:09.144761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.144739    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"
Oct 29 23:18:09.164486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.164455    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-dea9cb-serving-cert\""
Oct 29 23:18:09.165566 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.165546    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"]
Oct 29 23:18:09.219401 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.219364    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj6l6\" (UniqueName: \"kubernetes.io/projected/5603bb2b-7fee-4663-9f2d-9a550a7cca59-kube-api-access-bj6l6\") pod \"maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf\" (UID: \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"
Oct 29 23:18:09.219619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.219544    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5603bb2b-7fee-4663-9f2d-9a550a7cca59-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf\" (UID: \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"
Oct 29 23:18:09.219619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.219568    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5603bb2b-7fee-4663-9f2d-9a550a7cca59-maas-api-tls\") pod \"maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf\" (UID: \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"
Oct 29 23:18:09.219833 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.219777    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5603bb2b-7fee-4663-9f2d-9a550a7cca59-metrics-certs\") pod \"maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf\" (UID: \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"
Oct 29 23:18:09.220529 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.220490    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:09.321220 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.321191    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj6l6\" (UniqueName: \"kubernetes.io/projected/5603bb2b-7fee-4663-9f2d-9a550a7cca59-kube-api-access-bj6l6\") pod \"maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf\" (UID: \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"
Oct 29 23:18:09.321401 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.321304    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5603bb2b-7fee-4663-9f2d-9a550a7cca59-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf\" (UID: \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"
Oct 29 23:18:09.321401 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.321332    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5603bb2b-7fee-4663-9f2d-9a550a7cca59-maas-api-tls\") pod \"maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf\" (UID: \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"
Oct 29 23:18:09.321490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.321427    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5603bb2b-7fee-4663-9f2d-9a550a7cca59-metrics-certs\") pod \"maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf\" (UID: \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"
Oct 29 23:18:09.322020 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.321993    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5603bb2b-7fee-4663-9f2d-9a550a7cca59-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf\" (UID: \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"
Oct 29 23:18:09.323941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.323907    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5603bb2b-7fee-4663-9f2d-9a550a7cca59-maas-api-tls\") pod \"maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf\" (UID: \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"
Oct 29 23:18:09.324102 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.324077    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5603bb2b-7fee-4663-9f2d-9a550a7cca59-metrics-certs\") pod \"maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf\" (UID: \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"
Oct 29 23:18:09.333106 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.333080    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:09.344089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.344066    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj6l6\" (UniqueName: \"kubernetes.io/projected/5603bb2b-7fee-4663-9f2d-9a550a7cca59-kube-api-access-bj6l6\") pod \"maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf\" (UID: \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"
Oct 29 23:18:09.450964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.450879    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:09.462497 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.462465    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"
Oct 29 23:18:09.795481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:09.795448    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"]
Oct 29 23:18:09.796066 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:18:09.796034    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5603bb2b_7fee_4663_9f2d_9a550a7cca59.slice/crio-ff8cbd091e2e89c8e69e420fdf6c34fac856632b75221ad40d2b9c47874291f7 WatchSource:0}: Error finding container ff8cbd091e2e89c8e69e420fdf6c34fac856632b75221ad40d2b9c47874291f7: Status 404 returned error can't find the container with id ff8cbd091e2e89c8e69e420fdf6c34fac856632b75221ad40d2b9c47874291f7
Oct 29 23:18:10.479310 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:10.479184    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf" event={"ID":"5603bb2b-7fee-4663-9f2d-9a550a7cca59","Type":"ContainerStarted","Data":"fa13b71238612875c10616b8c30b2f67798dc1200bf78d2b888caba44bf64a6e"}
Oct 29 23:18:10.479310 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:10.479271    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf" event={"ID":"5603bb2b-7fee-4663-9f2d-9a550a7cca59","Type":"ContainerStarted","Data":"ff8cbd091e2e89c8e69e420fdf6c34fac856632b75221ad40d2b9c47874291f7"}
Oct 29 23:18:10.479310 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:10.479296    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"
Oct 29 23:18:10.525690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:10.525628    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf" podStartSLOduration=1.13435543 podStartE2EDuration="1.525613761s" totalImagesPullingTime="391.258331ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:10.5205834 +0000 UTC m=+2093.863532145" watchObservedRunningTime="2026-10-29 23:18:10.525613761 +0000 UTC m=+2093.868562505"
Oct 29 23:18:11.244432 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:11.244395    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:13.077864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:13.077827    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:14.784230 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:14.784198    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:16.497684 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:16.497654    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"
Oct 29 23:18:16.502855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:16.502831    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:17.371141 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:18:17.371103    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:18:18.369142 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:18.369106    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:20.470465 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:20.470423    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:22.173086 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:22.173050    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:23.875143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:23.875084    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:25.520829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:25.520780    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:27.220354 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:27.220320    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:28.840655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:28.840615    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:28.961685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:28.961650    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:29.088108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:29.088073    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:29.390771 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:29.390738    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:29.504120 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:29.504088    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:30.964318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:30.964279    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:32.659136 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:32.659102    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:34.297451 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:34.297413    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:36.070439 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:36.070405    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:37.746920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:37.746863    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:37.947615 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:37.947567    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"]
Oct 29 23:18:37.947862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:37.947788    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b" podUID="bf52a0e8-c739-47f8-9676-c5261e8d0f59" containerName="istio-proxy" containerID="cri-o://a60ccf3743cce0bf957b1afabc543f1a02f35a8ad2955c4b9bc4ed4b9b169e32" gracePeriod=30
Oct 29 23:18:39.446104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:39.446067    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:40.682097 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:40.682064    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"]
Oct 29 23:18:40.682549 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:40.682252    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn" podUID="db52cef6-d02d-439a-97ee-685acbad592d" containerName="istio-proxy" containerID="cri-o://a35be425041751b7e9fce7961305479a13fa689037c1b21fcecde5c3f9d1869b" gracePeriod=30
Oct 29 23:18:41.073255 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:41.073179    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:42.117863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:42.117827    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-98p7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:18:42.118326 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:42.117885    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b" podUID="bf52a0e8-c739-47f8-9676-c5261e8d0f59" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:18:42.436496 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:42.436419    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"]
Oct 29 23:18:42.436658 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:42.436638    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9" podUID="0a8f9e56-8bb0-4657-9539-f7b3aa80af15" containerName="istio-proxy" containerID="cri-o://163e4791592b6453ca1ef9720d3ebf5aef1f657b1af8a1165474d0d01f9ee166" gracePeriod=30
Oct 29 23:18:42.799789 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:42.799702    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:43.246459 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.246423    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 23:18:43.375115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.375022    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-podinfo\") pod \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") "
Oct 29 23:18:43.376343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.375440    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-credential-socket\") pod \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") "
Oct 29 23:18:43.376343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.375515    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf52a0e8-c739-47f8-9676-c5261e8d0f59-kube-api-access-rjz4z\" (UniqueName: \"kubernetes.io/projected/bf52a0e8-c739-47f8-9676-c5261e8d0f59-kube-api-access-rjz4z\") pod \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") "
Oct 29 23:18:43.376343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.375603    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-envoy\") pod \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") "
Oct 29 23:18:43.376343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.375660    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-data\") pod \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") "
Oct 29 23:18:43.376343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.375735    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-token\" (UniqueName: \"kubernetes.io/projected/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-token\") pod \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") "
Oct 29 23:18:43.376343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.375774    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-workload-socket\") pod \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") "
Oct 29 23:18:43.376343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.375957    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istiod-ca-cert\") pod \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") "
Oct 29 23:18:43.376343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.376002    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-workload-certs\") pod \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\" (UID: \"bf52a0e8-c739-47f8-9676-c5261e8d0f59\") "
Oct 29 23:18:43.378256 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.378185    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-workload-certs" pod "bf52a0e8-c739-47f8-9676-c5261e8d0f59" (UID: "bf52a0e8-c739-47f8-9676-c5261e8d0f59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:43.378452 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.378425    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-workload-socket" pod "bf52a0e8-c739-47f8-9676-c5261e8d0f59" (UID: "bf52a0e8-c739-47f8-9676-c5261e8d0f59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:43.378603 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.378585    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istiod-ca-cert" pod "bf52a0e8-c739-47f8-9676-c5261e8d0f59" (UID: "bf52a0e8-c739-47f8-9676-c5261e8d0f59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:18:43.378692 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.378656    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-data" pod "bf52a0e8-c739-47f8-9676-c5261e8d0f59" (UID: "bf52a0e8-c739-47f8-9676-c5261e8d0f59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:43.379473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.379438    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-token" pod "bf52a0e8-c739-47f8-9676-c5261e8d0f59" (UID: "bf52a0e8-c739-47f8-9676-c5261e8d0f59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:18:43.379473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.379448    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-credential-socket" pod "bf52a0e8-c739-47f8-9676-c5261e8d0f59" (UID: "bf52a0e8-c739-47f8-9676-c5261e8d0f59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:43.379644 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.379525    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-envoy" pod "bf52a0e8-c739-47f8-9676-c5261e8d0f59" (UID: "bf52a0e8-c739-47f8-9676-c5261e8d0f59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:43.381280 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.381252    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-podinfo" pod "bf52a0e8-c739-47f8-9676-c5261e8d0f59" (UID: "bf52a0e8-c739-47f8-9676-c5261e8d0f59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:18:43.382840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.382780    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf52a0e8-c739-47f8-9676-c5261e8d0f59-kube-api-access-rjz4z" pod "bf52a0e8-c739-47f8-9676-c5261e8d0f59" (UID: "bf52a0e8-c739-47f8-9676-c5261e8d0f59"). InnerVolumeSpecName "kube-api-access-rjz4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:18:43.480116 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.480085    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:43.480116 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.480113    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:43.480116 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.480123    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjz4z\" (UniqueName: \"kubernetes.io/projected/bf52a0e8-c739-47f8-9676-c5261e8d0f59-kube-api-access-rjz4z\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:43.480356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.480132    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:43.480356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.480141    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:43.480356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.480148    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:43.480356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.480155    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:43.480356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.480164    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf52a0e8-c739-47f8-9676-c5261e8d0f59-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:43.480356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.480172    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf52a0e8-c739-47f8-9676-c5261e8d0f59-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:43.887663 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.887626    5428 generic.go:356] "Generic (PLEG): container finished" podID="bf52a0e8-c739-47f8-9676-c5261e8d0f59" containerID="a60ccf3743cce0bf957b1afabc543f1a02f35a8ad2955c4b9bc4ed4b9b169e32" exitCode=0
Oct 29 23:18:43.887869 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.887679    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"
Oct 29 23:18:43.887869 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.887711    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b" event={"ID":"bf52a0e8-c739-47f8-9676-c5261e8d0f59","Type":"ContainerDied","Data":"a60ccf3743cce0bf957b1afabc543f1a02f35a8ad2955c4b9bc4ed4b9b169e32"}
Oct 29 23:18:43.887869 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.887778    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b" event={"ID":"bf52a0e8-c739-47f8-9676-c5261e8d0f59","Type":"ContainerDied","Data":"f91e539953b771366f040ed5d2a5ddc69becc1c4ea6653088eeadc4f32420785"}
Oct 29 23:18:43.887869 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.887825    5428 scope.go:122] "RemoveContainer" containerID="a60ccf3743cce0bf957b1afabc543f1a02f35a8ad2955c4b9bc4ed4b9b169e32"
Oct 29 23:18:43.903139 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.903122    5428 scope.go:122] "RemoveContainer" containerID="a60ccf3743cce0bf957b1afabc543f1a02f35a8ad2955c4b9bc4ed4b9b169e32"
Oct 29 23:18:43.903443 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:18:43.903422    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a60ccf3743cce0bf957b1afabc543f1a02f35a8ad2955c4b9bc4ed4b9b169e32\": container with ID starting with a60ccf3743cce0bf957b1afabc543f1a02f35a8ad2955c4b9bc4ed4b9b169e32 not found: ID does not exist" containerID="a60ccf3743cce0bf957b1afabc543f1a02f35a8ad2955c4b9bc4ed4b9b169e32"
Oct 29 23:18:43.903494 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.903451    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a60ccf3743cce0bf957b1afabc543f1a02f35a8ad2955c4b9bc4ed4b9b169e32"} err="failed to get container status \"a60ccf3743cce0bf957b1afabc543f1a02f35a8ad2955c4b9bc4ed4b9b169e32\": rpc error: code = NotFound desc = could not find container \"a60ccf3743cce0bf957b1afabc543f1a02f35a8ad2955c4b9bc4ed4b9b169e32\": container with ID starting with a60ccf3743cce0bf957b1afabc543f1a02f35a8ad2955c4b9bc4ed4b9b169e32 not found: ID does not exist"
Oct 29 23:18:43.954592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.954564    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"]
Oct 29 23:18:43.968588 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:43.968558    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-98p7b"]
Oct 29 23:18:44.479323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:44.479271    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:44.779186 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:44.779105    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf52a0e8-c739-47f8-9676-c5261e8d0f59" path="/var/lib/kubelet/pods/bf52a0e8-c739-47f8-9676-c5261e8d0f59/volumes"
Oct 29 23:18:45.912180 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:45.912147    5428 generic.go:356] "Generic (PLEG): container finished" podID="db52cef6-d02d-439a-97ee-685acbad592d" containerID="a35be425041751b7e9fce7961305479a13fa689037c1b21fcecde5c3f9d1869b" exitCode=0
Oct 29 23:18:45.912687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:45.912192    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn" event={"ID":"db52cef6-d02d-439a-97ee-685acbad592d","Type":"ContainerDied","Data":"a35be425041751b7e9fce7961305479a13fa689037c1b21fcecde5c3f9d1869b"}
Oct 29 23:18:45.968950 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:45.968884    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:18:46.076148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.076094    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:46.121863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.121832    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db52cef6-d02d-439a-97ee-685acbad592d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db52cef6-d02d-439a-97ee-685acbad592d-istiod-ca-cert\") pod \"db52cef6-d02d-439a-97ee-685acbad592d\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") "
Oct 29 23:18:46.122065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.121929    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db52cef6-d02d-439a-97ee-685acbad592d-istio-token\" (UniqueName: \"kubernetes.io/projected/db52cef6-d02d-439a-97ee-685acbad592d-istio-token\") pod \"db52cef6-d02d-439a-97ee-685acbad592d\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") "
Oct 29 23:18:46.122144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.122069    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-workload-socket\") pod \"db52cef6-d02d-439a-97ee-685acbad592d\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") "
Oct 29 23:18:46.122144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.122112    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-istio-envoy\") pod \"db52cef6-d02d-439a-97ee-685acbad592d\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") "
Oct 29 23:18:46.122262 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.122178    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db52cef6-d02d-439a-97ee-685acbad592d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db52cef6-d02d-439a-97ee-685acbad592d-istio-podinfo\") pod \"db52cef6-d02d-439a-97ee-685acbad592d\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") "
Oct 29 23:18:46.122262 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.122214    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-istio-data\") pod \"db52cef6-d02d-439a-97ee-685acbad592d\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") "
Oct 29 23:18:46.122387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.122246    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db52cef6-d02d-439a-97ee-685acbad592d-istiod-ca-cert" pod "db52cef6-d02d-439a-97ee-685acbad592d" (UID: "db52cef6-d02d-439a-97ee-685acbad592d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:18:46.122387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.122348    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db52cef6-d02d-439a-97ee-685acbad592d-kube-api-access-wvs2x\" (UniqueName: \"kubernetes.io/projected/db52cef6-d02d-439a-97ee-685acbad592d-kube-api-access-wvs2x\") pod \"db52cef6-d02d-439a-97ee-685acbad592d\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") "
Oct 29 23:18:46.122387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.122373    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-workload-socket" pod "db52cef6-d02d-439a-97ee-685acbad592d" (UID: "db52cef6-d02d-439a-97ee-685acbad592d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:46.122572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.122397    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-workload-certs\") pod \"db52cef6-d02d-439a-97ee-685acbad592d\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") "
Oct 29 23:18:46.122572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.122510    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-credential-socket\") pod \"db52cef6-d02d-439a-97ee-685acbad592d\" (UID: \"db52cef6-d02d-439a-97ee-685acbad592d\") "
Oct 29 23:18:46.122572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.122558    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-istio-data" pod "db52cef6-d02d-439a-97ee-685acbad592d" (UID: "db52cef6-d02d-439a-97ee-685acbad592d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:46.122774 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.122730    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-workload-certs" pod "db52cef6-d02d-439a-97ee-685acbad592d" (UID: "db52cef6-d02d-439a-97ee-685acbad592d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:46.122881 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.122832    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-credential-socket" pod "db52cef6-d02d-439a-97ee-685acbad592d" (UID: "db52cef6-d02d-439a-97ee-685acbad592d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:46.123405 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.123371    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:46.123405 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.123402    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:46.123616 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.123419    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db52cef6-d02d-439a-97ee-685acbad592d-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:46.123616 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.123435    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:46.123616 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.123448    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:46.124509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.124481    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db52cef6-d02d-439a-97ee-685acbad592d-istio-token" pod "db52cef6-d02d-439a-97ee-685acbad592d" (UID: "db52cef6-d02d-439a-97ee-685acbad592d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:18:46.124646 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.124626    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db52cef6-d02d-439a-97ee-685acbad592d-istio-podinfo" pod "db52cef6-d02d-439a-97ee-685acbad592d" (UID: "db52cef6-d02d-439a-97ee-685acbad592d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:18:46.124952 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.124934    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-istio-envoy" pod "db52cef6-d02d-439a-97ee-685acbad592d" (UID: "db52cef6-d02d-439a-97ee-685acbad592d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:46.124952 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.124942    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db52cef6-d02d-439a-97ee-685acbad592d-kube-api-access-wvs2x" pod "db52cef6-d02d-439a-97ee-685acbad592d" (UID: "db52cef6-d02d-439a-97ee-685acbad592d"). InnerVolumeSpecName "kube-api-access-wvs2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:18:46.224587 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.224505    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db52cef6-d02d-439a-97ee-685acbad592d-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:46.224587 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.224534    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db52cef6-d02d-439a-97ee-685acbad592d-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:46.224587 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.224543    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db52cef6-d02d-439a-97ee-685acbad592d-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:46.224587 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.224552    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvs2x\" (UniqueName: \"kubernetes.io/projected/db52cef6-d02d-439a-97ee-685acbad592d-kube-api-access-wvs2x\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:46.922129 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.922101    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"
Oct 29 23:18:46.922564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.922130    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn" event={"ID":"db52cef6-d02d-439a-97ee-685acbad592d","Type":"ContainerDied","Data":"0a5aa95b1e702fab028998b493bb0479fb5e1321c93c792b224f837db7f161c0"}
Oct 29 23:18:46.922564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.922198    5428 scope.go:122] "RemoveContainer" containerID="a35be425041751b7e9fce7961305479a13fa689037c1b21fcecde5c3f9d1869b"
Oct 29 23:18:46.989058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:46.989028    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"]
Oct 29 23:18:47.001313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.001288    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-q9twn"]
Oct 29 23:18:47.370146 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:18:47.370105    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:18:47.736180 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.736154    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:18:47.834924 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.834882    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:47.850026 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.849993    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-kube-api-access-95wft\" (UniqueName: \"kubernetes.io/projected/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-kube-api-access-95wft\") pod \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") "
Oct 29 23:18:47.850317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.850288    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-token\" (UniqueName: \"kubernetes.io/projected/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-token\") pod \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") "
Oct 29 23:18:47.850537 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.850351    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-credential-socket\") pod \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") "
Oct 29 23:18:47.850537 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.850446    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-podinfo\") pod \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") "
Oct 29 23:18:47.850537 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.850477    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-envoy\") pod \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") "
Oct 29 23:18:47.850736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.850557    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-workload-certs\") pod \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") "
Oct 29 23:18:47.850736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.850655    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-workload-socket\") pod \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") "
Oct 29 23:18:47.850736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.850713    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istiod-ca-cert\") pod \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") "
Oct 29 23:18:47.850922 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.850908    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-data\") pod \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\" (UID: \"0a8f9e56-8bb0-4657-9539-f7b3aa80af15\") "
Oct 29 23:18:47.852522 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.851385    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-workload-socket" pod "0a8f9e56-8bb0-4657-9539-f7b3aa80af15" (UID: "0a8f9e56-8bb0-4657-9539-f7b3aa80af15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:47.852522 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.851842    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-workload-certs" pod "0a8f9e56-8bb0-4657-9539-f7b3aa80af15" (UID: "0a8f9e56-8bb0-4657-9539-f7b3aa80af15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:47.852522 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.851927    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:47.852522 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.852142    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-data" pod "0a8f9e56-8bb0-4657-9539-f7b3aa80af15" (UID: "0a8f9e56-8bb0-4657-9539-f7b3aa80af15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:47.852522 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.852212    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istiod-ca-cert" pod "0a8f9e56-8bb0-4657-9539-f7b3aa80af15" (UID: "0a8f9e56-8bb0-4657-9539-f7b3aa80af15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:18:47.853119 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.853040    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-kube-api-access-95wft" pod "0a8f9e56-8bb0-4657-9539-f7b3aa80af15" (UID: "0a8f9e56-8bb0-4657-9539-f7b3aa80af15"). InnerVolumeSpecName "kube-api-access-95wft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:18:47.853458 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.853305    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-token" pod "0a8f9e56-8bb0-4657-9539-f7b3aa80af15" (UID: "0a8f9e56-8bb0-4657-9539-f7b3aa80af15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:18:47.853458 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.853388    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-podinfo" pod "0a8f9e56-8bb0-4657-9539-f7b3aa80af15" (UID: "0a8f9e56-8bb0-4657-9539-f7b3aa80af15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:18:47.853631 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.853493    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-envoy" pod "0a8f9e56-8bb0-4657-9539-f7b3aa80af15" (UID: "0a8f9e56-8bb0-4657-9539-f7b3aa80af15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:47.853776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.853747    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-credential-socket" pod "0a8f9e56-8bb0-4657-9539-f7b3aa80af15" (UID: "0a8f9e56-8bb0-4657-9539-f7b3aa80af15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:18:47.938549 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.938512    5428 generic.go:356] "Generic (PLEG): container finished" podID="0a8f9e56-8bb0-4657-9539-f7b3aa80af15" containerID="163e4791592b6453ca1ef9720d3ebf5aef1f657b1af8a1165474d0d01f9ee166" exitCode=0
Oct 29 23:18:47.939144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.938571    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"
Oct 29 23:18:47.939144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.938576    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9" event={"ID":"0a8f9e56-8bb0-4657-9539-f7b3aa80af15","Type":"ContainerDied","Data":"163e4791592b6453ca1ef9720d3ebf5aef1f657b1af8a1165474d0d01f9ee166"}
Oct 29 23:18:47.939144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.938631    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9" event={"ID":"0a8f9e56-8bb0-4657-9539-f7b3aa80af15","Type":"ContainerDied","Data":"77f5beec63d4615b272b2830718fcb4207f41577bf6520c9fbb30a3cdda1a5b8"}
Oct 29 23:18:47.939144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.938657    5428 scope.go:122] "RemoveContainer" containerID="163e4791592b6453ca1ef9720d3ebf5aef1f657b1af8a1165474d0d01f9ee166"
Oct 29 23:18:47.953078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.953059    5428 scope.go:122] "RemoveContainer" containerID="163e4791592b6453ca1ef9720d3ebf5aef1f657b1af8a1165474d0d01f9ee166"
Oct 29 23:18:47.953439 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:18:47.953387    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163e4791592b6453ca1ef9720d3ebf5aef1f657b1af8a1165474d0d01f9ee166\": container with ID starting with 163e4791592b6453ca1ef9720d3ebf5aef1f657b1af8a1165474d0d01f9ee166 not found: ID does not exist" containerID="163e4791592b6453ca1ef9720d3ebf5aef1f657b1af8a1165474d0d01f9ee166"
Oct 29 23:18:47.953544 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.953450    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163e4791592b6453ca1ef9720d3ebf5aef1f657b1af8a1165474d0d01f9ee166"} err="failed to get container status \"163e4791592b6453ca1ef9720d3ebf5aef1f657b1af8a1165474d0d01f9ee166\": rpc error: code = NotFound desc = could not find container \"163e4791592b6453ca1ef9720d3ebf5aef1f657b1af8a1165474d0d01f9ee166\": container with ID starting with 163e4791592b6453ca1ef9720d3ebf5aef1f657b1af8a1165474d0d01f9ee166 not found: ID does not exist"
Oct 29 23:18:47.953960 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.953940    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95wft\" (UniqueName: \"kubernetes.io/projected/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-kube-api-access-95wft\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:47.953960 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.953959    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:47.954134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.953970    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:47.954134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.953980    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:47.954134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.953988    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:47.954134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.953997    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:47.954134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.954005    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:47.954134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.954012    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8f9e56-8bb0-4657-9539-f7b3aa80af15-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:18:47.994604 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.994569    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"]
Oct 29 23:18:47.999256 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:47.999232    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-6m5v9"]
Oct 29 23:18:48.779527 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:48.779497    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a8f9e56-8bb0-4657-9539-f7b3aa80af15" path="/var/lib/kubelet/pods/0a8f9e56-8bb0-4657-9539-f7b3aa80af15/volumes"
Oct 29 23:18:48.780053 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:48.780038    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db52cef6-d02d-439a-97ee-685acbad592d" path="/var/lib/kubelet/pods/db52cef6-d02d-439a-97ee-685acbad592d/volumes"
Oct 29 23:18:49.792979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:49.792949    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:51.578428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:51.578383    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:51.877933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:51.877899    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:52.972196 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:52.972165    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:53.427358 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:53.427328    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:53.593227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:53.593183    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:54.065226 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:54.065195    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:55.687972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:55.687936    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:57.502436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:57.502403    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:57.632430 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:57.627376    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:59.083703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:59.083674    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:59.238987 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:18:59.238954    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:00.088637 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:00.088578    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:01.867333 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:01.867282    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:03.585774 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:03.585738    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:05.308742 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:05.308710    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:07.110306 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:07.110240    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:10.116034 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:10.115954    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:12.643786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:12.643744    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:14.885782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:14.885742    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:16.190613 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:16.190583    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:16.684511 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:16.684477    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:17.369441 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:19:17.369398    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:19:18.335083 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:18.335047    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"]
Oct 29 23:19:18.335293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:18.335264    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg" podUID="a0b7e549-f379-4461-ad77-c3ce9c1d00fc" containerName="maas-api" containerID="cri-o://f06a9e3943737290e05fcf37a314c7aa950315f429247d610a4e553f03bb9178" gracePeriod=30
Oct 29 23:19:18.400263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:18.400225    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:18.802742 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:18.802714    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"
Oct 29 23:19:18.972582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:18.972492    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-metrics-certs\") pod \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\" (UID: \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\") "
Oct 29 23:19:18.972751 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:18.972607    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-openshift-service-ca\") pod \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\" (UID: \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\") "
Oct 29 23:19:18.972879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:18.972862    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-maas-api-tls\") pod \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\" (UID: \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\") "
Oct 29 23:19:18.972942 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:18.972913    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-kube-api-access-xtb9t\" (UniqueName: \"kubernetes.io/projected/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-kube-api-access-xtb9t\") pod \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\" (UID: \"a0b7e549-f379-4461-ad77-c3ce9c1d00fc\") "
Oct 29 23:19:18.973004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:18.972970    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-openshift-service-ca" pod "a0b7e549-f379-4461-ad77-c3ce9c1d00fc" (UID: "a0b7e549-f379-4461-ad77-c3ce9c1d00fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:19:18.973616 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:18.973594    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:19:18.974715 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:18.974692    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-metrics-certs" pod "a0b7e549-f379-4461-ad77-c3ce9c1d00fc" (UID: "a0b7e549-f379-4461-ad77-c3ce9c1d00fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:19:18.975003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:18.974979    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-maas-api-tls" pod "a0b7e549-f379-4461-ad77-c3ce9c1d00fc" (UID: "a0b7e549-f379-4461-ad77-c3ce9c1d00fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:19:18.975213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:18.975192    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-kube-api-access-xtb9t" pod "a0b7e549-f379-4461-ad77-c3ce9c1d00fc" (UID: "a0b7e549-f379-4461-ad77-c3ce9c1d00fc"). InnerVolumeSpecName "kube-api-access-xtb9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:19:19.074763 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:19.074727    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtb9t\" (UniqueName: \"kubernetes.io/projected/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-kube-api-access-xtb9t\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:19:19.074763 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:19.074760    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:19:19.074763 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:19.074769    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b7e549-f379-4461-ad77-c3ce9c1d00fc-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:19:19.234499 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:19.234418    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:19.297691 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:19.297654    5428 generic.go:356] "Generic (PLEG): container finished" podID="a0b7e549-f379-4461-ad77-c3ce9c1d00fc" containerID="f06a9e3943737290e05fcf37a314c7aa950315f429247d610a4e553f03bb9178" exitCode=0
Oct 29 23:19:19.297914 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:19.297720    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"
Oct 29 23:19:19.297914 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:19.297743    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg" event={"ID":"a0b7e549-f379-4461-ad77-c3ce9c1d00fc","Type":"ContainerDied","Data":"f06a9e3943737290e05fcf37a314c7aa950315f429247d610a4e553f03bb9178"}
Oct 29 23:19:19.297914 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:19.297839    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg" event={"ID":"a0b7e549-f379-4461-ad77-c3ce9c1d00fc","Type":"ContainerDied","Data":"0dbd72b558579f10061ec978466b4e1869bfed63d435245765d7c50ca89394f0"}
Oct 29 23:19:19.297914 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:19.297862    5428 scope.go:122] "RemoveContainer" containerID="f06a9e3943737290e05fcf37a314c7aa950315f429247d610a4e553f03bb9178"
Oct 29 23:19:19.310387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:19.310366    5428 scope.go:122] "RemoveContainer" containerID="f06a9e3943737290e05fcf37a314c7aa950315f429247d610a4e553f03bb9178"
Oct 29 23:19:19.310701 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:19:19.310680    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f06a9e3943737290e05fcf37a314c7aa950315f429247d610a4e553f03bb9178\": container with ID starting with f06a9e3943737290e05fcf37a314c7aa950315f429247d610a4e553f03bb9178 not found: ID does not exist" containerID="f06a9e3943737290e05fcf37a314c7aa950315f429247d610a4e553f03bb9178"
Oct 29 23:19:19.310756 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:19.310709    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f06a9e3943737290e05fcf37a314c7aa950315f429247d610a4e553f03bb9178"} err="failed to get container status \"f06a9e3943737290e05fcf37a314c7aa950315f429247d610a4e553f03bb9178\": rpc error: code = NotFound desc = could not find container \"f06a9e3943737290e05fcf37a314c7aa950315f429247d610a4e553f03bb9178\": container with ID starting with f06a9e3943737290e05fcf37a314c7aa950315f429247d610a4e553f03bb9178 not found: ID does not exist"
Oct 29 23:19:19.350941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:19.350914    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"]
Oct 29 23:19:19.355225 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:19.355204    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-96cd49-6c6689d55-r8dhg"]
Oct 29 23:19:19.379576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:19.379553    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:19.513860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:19.513763    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:19.903757 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:19.903725    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:20.779347 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:20.779316    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b7e549-f379-4461-ad77-c3ce9c1d00fc" path="/var/lib/kubelet/pods/a0b7e549-f379-4461-ad77-c3ce9c1d00fc/volumes"
Oct 29 23:19:21.488876 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:21.488841    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:23.493768 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:23.493733    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:24.959870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:24.959833    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:25.649635 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:25.649598    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:27.892143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:27.892064    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:28.929068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:28.929032    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:29.510476 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:29.510440    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:29.798125 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:29.798039    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:30.656670 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:30.656633    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:31.113330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:31.113253    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:31.237348 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:31.237297    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:31.889370 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:31.889339    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:33.861830 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:33.861785    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:34.479390 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:34.479357    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:34.594536 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:34.594498    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:36.028720 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:36.028686    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:36.149598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:36.149564    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:36.286289 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:36.286203    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:36.410695 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:36.410660    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:38.009606 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:38.009571    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:39.786422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:39.786391    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:41.565538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:41.565501    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:43.250061 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:43.250026    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:43.375812 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:43.375781    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:43.499670 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:43.499638    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:45.246388 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:45.246347    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:47.002470 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:47.002434    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:47.370836 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:19:47.370763    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:19:48.880692 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:48.880659    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:50.656478 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:50.656443    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:52.396735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:52.396697    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:54.185437 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:54.185400    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:55.894316 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:55.894283    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"]
Oct 29 23:19:55.894792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:55.894497    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf" podUID="1b154cd7-0f58-4506-b7ac-2eed207e4a0b" containerName="istio-proxy" containerID="cri-o://c7e5ffafe624e7674aed5430da116b976d2d15375838c0e4d94ee33cc9421d75" gracePeriod=30
Oct 29 23:19:55.974946 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:55.974916    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:57.885938 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:57.885901    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:59.714894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:19:59.714770    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:01.223683 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.223647    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:20:01.315724 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.315672    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-podinfo\") pod \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") "
Oct 29 23:20:01.315974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.315951    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-credential-socket\") pod \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") "
Oct 29 23:20:01.316066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.316030    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-kube-api-access-kwcnb\" (UniqueName: \"kubernetes.io/projected/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-kube-api-access-kwcnb\") pod \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") "
Oct 29 23:20:01.316141 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.316076    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-envoy\") pod \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") "
Oct 29 23:20:01.316141 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.316133    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-workload-certs\") pod \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") "
Oct 29 23:20:01.316257 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.316162    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istiod-ca-cert\") pod \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") "
Oct 29 23:20:01.316257 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.316219    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-data\") pod \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") "
Oct 29 23:20:01.316372 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.316256    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-workload-socket\") pod \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") "
Oct 29 23:20:01.316372 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.316303    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-credential-socket" pod "1b154cd7-0f58-4506-b7ac-2eed207e4a0b" (UID: "1b154cd7-0f58-4506-b7ac-2eed207e4a0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:20:01.316499 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.316380    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-token\" (UniqueName: \"kubernetes.io/projected/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-token\") pod \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\" (UID: \"1b154cd7-0f58-4506-b7ac-2eed207e4a0b\") "
Oct 29 23:20:01.316563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.316519    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-workload-certs" pod "1b154cd7-0f58-4506-b7ac-2eed207e4a0b" (UID: "1b154cd7-0f58-4506-b7ac-2eed207e4a0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:20:01.317140 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.317114    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:20:01.317283 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.317143    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:20:01.318854 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.318770    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istiod-ca-cert" pod "1b154cd7-0f58-4506-b7ac-2eed207e4a0b" (UID: "1b154cd7-0f58-4506-b7ac-2eed207e4a0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:20:01.319046 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.319015    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-workload-socket" pod "1b154cd7-0f58-4506-b7ac-2eed207e4a0b" (UID: "1b154cd7-0f58-4506-b7ac-2eed207e4a0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:20:01.319157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.319106    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-envoy" pod "1b154cd7-0f58-4506-b7ac-2eed207e4a0b" (UID: "1b154cd7-0f58-4506-b7ac-2eed207e4a0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:20:01.322792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.322754    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-kube-api-access-kwcnb" pod "1b154cd7-0f58-4506-b7ac-2eed207e4a0b" (UID: "1b154cd7-0f58-4506-b7ac-2eed207e4a0b"). InnerVolumeSpecName "kube-api-access-kwcnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:20:01.322925 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.322868    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-podinfo" pod "1b154cd7-0f58-4506-b7ac-2eed207e4a0b" (UID: "1b154cd7-0f58-4506-b7ac-2eed207e4a0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:20:01.323939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.323907    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-data" pod "1b154cd7-0f58-4506-b7ac-2eed207e4a0b" (UID: "1b154cd7-0f58-4506-b7ac-2eed207e4a0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:20:01.324843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.324133    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-token" pod "1b154cd7-0f58-4506-b7ac-2eed207e4a0b" (UID: "1b154cd7-0f58-4506-b7ac-2eed207e4a0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:20:01.418597 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.418564    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:20:01.418597 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.418593    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:20:01.418597 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.418603    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwcnb\" (UniqueName: \"kubernetes.io/projected/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-kube-api-access-kwcnb\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:20:01.418885 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.418612    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:20:01.418885 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.418621    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:20:01.418885 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.418629    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:20:01.418885 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.418636    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b154cd7-0f58-4506-b7ac-2eed207e4a0b-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:20:01.529359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.529272    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:01.666834 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.666781    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:01.723824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.723643    5428 generic.go:356] "Generic (PLEG): container finished" podID="1b154cd7-0f58-4506-b7ac-2eed207e4a0b" containerID="c7e5ffafe624e7674aed5430da116b976d2d15375838c0e4d94ee33cc9421d75" exitCode=0
Oct 29 23:20:01.723824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.723691    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf" event={"ID":"1b154cd7-0f58-4506-b7ac-2eed207e4a0b","Type":"ContainerDied","Data":"c7e5ffafe624e7674aed5430da116b976d2d15375838c0e4d94ee33cc9421d75"}
Oct 29 23:20:01.723824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.723741    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf" event={"ID":"1b154cd7-0f58-4506-b7ac-2eed207e4a0b","Type":"ContainerDied","Data":"c1dcd1a76817397bac6191a73c2be131254c9ca5ae188d6b495d971934aed15a"}
Oct 29 23:20:01.723824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.723770    5428 scope.go:122] "RemoveContainer" containerID="c7e5ffafe624e7674aed5430da116b976d2d15375838c0e4d94ee33cc9421d75"
Oct 29 23:20:01.724245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.723936    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"
Oct 29 23:20:01.761730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.761644    5428 scope.go:122] "RemoveContainer" containerID="c7e5ffafe624e7674aed5430da116b976d2d15375838c0e4d94ee33cc9421d75"
Oct 29 23:20:01.764531 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:20:01.764449    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e5ffafe624e7674aed5430da116b976d2d15375838c0e4d94ee33cc9421d75\": container with ID starting with c7e5ffafe624e7674aed5430da116b976d2d15375838c0e4d94ee33cc9421d75 not found: ID does not exist" containerID="c7e5ffafe624e7674aed5430da116b976d2d15375838c0e4d94ee33cc9421d75"
Oct 29 23:20:01.764531 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.764497    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e5ffafe624e7674aed5430da116b976d2d15375838c0e4d94ee33cc9421d75"} err="failed to get container status \"c7e5ffafe624e7674aed5430da116b976d2d15375838c0e4d94ee33cc9421d75\": rpc error: code = NotFound desc = could not find container \"c7e5ffafe624e7674aed5430da116b976d2d15375838c0e4d94ee33cc9421d75\": container with ID starting with c7e5ffafe624e7674aed5430da116b976d2d15375838c0e4d94ee33cc9421d75 not found: ID does not exist"
Oct 29 23:20:01.796521 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.796397    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"]
Oct 29 23:20:01.805538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:01.805487    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-x7qnf"]
Oct 29 23:20:02.780312 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:02.780279    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b154cd7-0f58-4506-b7ac-2eed207e4a0b" path="/var/lib/kubelet/pods/1b154cd7-0f58-4506-b7ac-2eed207e4a0b/volumes"
Oct 29 23:20:03.731855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:03.731816    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:03.853875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:03.853839    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:05.578581 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:05.578549    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:07.484319 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:07.484284    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:09.244400 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:09.244367    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:10.096049 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:10.096015    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:10.220682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:10.220646    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:10.432412 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:10.432383    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:11.244599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:11.244568    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:13.198790 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:13.198752    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:15.191782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:15.191749    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:17.044525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:17.044493    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:17.187066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:17.187032    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:17.310388 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:17.310287    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:17.369126 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:20:17.369091    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:20:18.993525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:18.993488    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:20.823663 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:20.823629    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:21.377701 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:21.377647    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4"]
Oct 29 23:20:21.413058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:21.413026    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj"]
Oct 29 23:20:21.528080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:21.528041    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:21.640689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:21.640589    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:22.616965 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:22.616930    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:24.437680 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:24.437637    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:26.350623 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.350592    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:26.378885 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.378766    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4" podUID="db877578-1ef2-4a99-b585-0ea0504a9613" containerName="payload-processing" containerID="cri-o://647140312bb15b4fc4adcea002553e23ceedcd070c9b6ef1a49c4ddf3ce7388b" gracePeriod=25
Oct 29 23:20:26.413646 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.413404    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj" podUID="b42c20e6-b20d-4380-a772-b06a9b3d3592" containerName="payload-pre-processing" containerID="cri-o://6456f2646a10d5cef465fa484b53076e998ccd2c9523779f3df970dc8574fa1b" gracePeriod=25
Oct 29 23:20:26.709664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.709640    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4"
Oct 29 23:20:26.759361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.759328    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj"
Oct 29 23:20:26.765884 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.765853    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db877578-1ef2-4a99-b585-0ea0504a9613-kube-api-access-blrmx\" (UniqueName: \"kubernetes.io/projected/db877578-1ef2-4a99-b585-0ea0504a9613-kube-api-access-blrmx\") pod \"db877578-1ef2-4a99-b585-0ea0504a9613\" (UID: \"db877578-1ef2-4a99-b585-0ea0504a9613\") "
Oct 29 23:20:26.766035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.765966    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db877578-1ef2-4a99-b585-0ea0504a9613-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db877578-1ef2-4a99-b585-0ea0504a9613-plugins-config-volume\") pod \"db877578-1ef2-4a99-b585-0ea0504a9613\" (UID: \"db877578-1ef2-4a99-b585-0ea0504a9613\") "
Oct 29 23:20:26.766378 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.766352    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db877578-1ef2-4a99-b585-0ea0504a9613-plugins-config-volume" pod "db877578-1ef2-4a99-b585-0ea0504a9613" (UID: "db877578-1ef2-4a99-b585-0ea0504a9613"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:20:26.766920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.766897    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db877578-1ef2-4a99-b585-0ea0504a9613-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:20:26.768245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.768223    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db877578-1ef2-4a99-b585-0ea0504a9613-kube-api-access-blrmx" pod "db877578-1ef2-4a99-b585-0ea0504a9613" (UID: "db877578-1ef2-4a99-b585-0ea0504a9613"). InnerVolumeSpecName "kube-api-access-blrmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:20:26.867727 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.867698    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b42c20e6-b20d-4380-a772-b06a9b3d3592-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b42c20e6-b20d-4380-a772-b06a9b3d3592-plugins-config-volume\") pod \"b42c20e6-b20d-4380-a772-b06a9b3d3592\" (UID: \"b42c20e6-b20d-4380-a772-b06a9b3d3592\") "
Oct 29 23:20:26.867907 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.867787    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42c20e6-b20d-4380-a772-b06a9b3d3592-kube-api-access-kbsjz\" (UniqueName: \"kubernetes.io/projected/b42c20e6-b20d-4380-a772-b06a9b3d3592-kube-api-access-kbsjz\") pod \"b42c20e6-b20d-4380-a772-b06a9b3d3592\" (UID: \"b42c20e6-b20d-4380-a772-b06a9b3d3592\") "
Oct 29 23:20:26.868116 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.868093    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42c20e6-b20d-4380-a772-b06a9b3d3592-plugins-config-volume" pod "b42c20e6-b20d-4380-a772-b06a9b3d3592" (UID: "b42c20e6-b20d-4380-a772-b06a9b3d3592"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:20:26.868387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.868371    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blrmx\" (UniqueName: \"kubernetes.io/projected/db877578-1ef2-4a99-b585-0ea0504a9613-kube-api-access-blrmx\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:20:26.868435 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.868389    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b42c20e6-b20d-4380-a772-b06a9b3d3592-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:20:26.869783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.869763    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42c20e6-b20d-4380-a772-b06a9b3d3592-kube-api-access-kbsjz" pod "b42c20e6-b20d-4380-a772-b06a9b3d3592" (UID: "b42c20e6-b20d-4380-a772-b06a9b3d3592"). InnerVolumeSpecName "kube-api-access-kbsjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:20:26.952055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.951986    5428 generic.go:356] "Generic (PLEG): container finished" podID="db877578-1ef2-4a99-b585-0ea0504a9613" containerID="647140312bb15b4fc4adcea002553e23ceedcd070c9b6ef1a49c4ddf3ce7388b" exitCode=0
Oct 29 23:20:26.952055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.952035    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4"
Oct 29 23:20:26.952276 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.952076    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4" event={"ID":"db877578-1ef2-4a99-b585-0ea0504a9613","Type":"ContainerDied","Data":"647140312bb15b4fc4adcea002553e23ceedcd070c9b6ef1a49c4ddf3ce7388b"}
Oct 29 23:20:26.952276 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.952140    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4" event={"ID":"db877578-1ef2-4a99-b585-0ea0504a9613","Type":"ContainerDied","Data":"4dd9e228e57ce41999be85a718a0a05f6a938c691a72164bf56ae4242b6c756b"}
Oct 29 23:20:26.952276 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.952170    5428 scope.go:122] "RemoveContainer" containerID="647140312bb15b4fc4adcea002553e23ceedcd070c9b6ef1a49c4ddf3ce7388b"
Oct 29 23:20:26.953576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.953553    5428 generic.go:356] "Generic (PLEG): container finished" podID="b42c20e6-b20d-4380-a772-b06a9b3d3592" containerID="6456f2646a10d5cef465fa484b53076e998ccd2c9523779f3df970dc8574fa1b" exitCode=0
Oct 29 23:20:26.953685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.953605    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj" event={"ID":"b42c20e6-b20d-4380-a772-b06a9b3d3592","Type":"ContainerDied","Data":"6456f2646a10d5cef465fa484b53076e998ccd2c9523779f3df970dc8574fa1b"}
Oct 29 23:20:26.953685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.953635    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj" event={"ID":"b42c20e6-b20d-4380-a772-b06a9b3d3592","Type":"ContainerDied","Data":"a24dfed56b54feb337b7e73fb50235934e0574edea6340b0e03901c52a8c80ac"}
Oct 29 23:20:26.953685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.953606    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj"
Oct 29 23:20:26.963579 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.963563    5428 scope.go:122] "RemoveContainer" containerID="647140312bb15b4fc4adcea002553e23ceedcd070c9b6ef1a49c4ddf3ce7388b"
Oct 29 23:20:26.963783 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:20:26.963764    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"647140312bb15b4fc4adcea002553e23ceedcd070c9b6ef1a49c4ddf3ce7388b\": container with ID starting with 647140312bb15b4fc4adcea002553e23ceedcd070c9b6ef1a49c4ddf3ce7388b not found: ID does not exist" containerID="647140312bb15b4fc4adcea002553e23ceedcd070c9b6ef1a49c4ddf3ce7388b"
Oct 29 23:20:26.963853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.963794    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"647140312bb15b4fc4adcea002553e23ceedcd070c9b6ef1a49c4ddf3ce7388b"} err="failed to get container status \"647140312bb15b4fc4adcea002553e23ceedcd070c9b6ef1a49c4ddf3ce7388b\": rpc error: code = NotFound desc = could not find container \"647140312bb15b4fc4adcea002553e23ceedcd070c9b6ef1a49c4ddf3ce7388b\": container with ID starting with 647140312bb15b4fc4adcea002553e23ceedcd070c9b6ef1a49c4ddf3ce7388b not found: ID does not exist"
Oct 29 23:20:26.963853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.963845    5428 scope.go:122] "RemoveContainer" containerID="6456f2646a10d5cef465fa484b53076e998ccd2c9523779f3df970dc8574fa1b"
Oct 29 23:20:26.969829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.969794    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbsjz\" (UniqueName: \"kubernetes.io/projected/b42c20e6-b20d-4380-a772-b06a9b3d3592-kube-api-access-kbsjz\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:20:26.973393 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.973373    5428 scope.go:122] "RemoveContainer" containerID="6456f2646a10d5cef465fa484b53076e998ccd2c9523779f3df970dc8574fa1b"
Oct 29 23:20:26.973635 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:20:26.973619    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6456f2646a10d5cef465fa484b53076e998ccd2c9523779f3df970dc8574fa1b\": container with ID starting with 6456f2646a10d5cef465fa484b53076e998ccd2c9523779f3df970dc8574fa1b not found: ID does not exist" containerID="6456f2646a10d5cef465fa484b53076e998ccd2c9523779f3df970dc8574fa1b"
Oct 29 23:20:26.973685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:26.973641    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6456f2646a10d5cef465fa484b53076e998ccd2c9523779f3df970dc8574fa1b"} err="failed to get container status \"6456f2646a10d5cef465fa484b53076e998ccd2c9523779f3df970dc8574fa1b\": rpc error: code = NotFound desc = could not find container \"6456f2646a10d5cef465fa484b53076e998ccd2c9523779f3df970dc8574fa1b\": container with ID starting with 6456f2646a10d5cef465fa484b53076e998ccd2c9523779f3df970dc8574fa1b not found: ID does not exist"
Oct 29 23:20:27.009559 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:27.009529    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj"]
Oct 29 23:20:27.013998 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:27.013977    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7a3c1a-7d46cf9d7-q6ftj"]
Oct 29 23:20:27.047712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:27.047682    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4"]
Oct 29 23:20:27.051752 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:27.051732    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7a3c1a-7b9d57bf66-wlrr4"]
Oct 29 23:20:28.236285 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:28.236251    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:28.778914 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:28.778883    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b42c20e6-b20d-4380-a772-b06a9b3d3592" path="/var/lib/kubelet/pods/b42c20e6-b20d-4380-a772-b06a9b3d3592/volumes"
Oct 29 23:20:28.779406 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:28.779392    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db877578-1ef2-4a99-b585-0ea0504a9613" path="/var/lib/kubelet/pods/db877578-1ef2-4a99-b585-0ea0504a9613/volumes"
Oct 29 23:20:29.048099 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:29.048014    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:29.645023 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:29.644981    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:30.411348 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:30.411317    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:32.304465 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:32.304433    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:34.321122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:34.321084    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:34.731617 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:34.731580    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:35.978392 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:35.978357    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:37.395654 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:37.395621    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:37.848046 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:37.847962    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:38.026876 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:38.026840    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:39.273823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:39.237315    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:39.369176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:39.369081    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:41.946818 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:41.946773    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:43.913238 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:43.913203    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:45.931671 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:45.931632    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:47.372092 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:20:47.372047    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:20:47.651576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:47.651489    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:49.491523 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:49.491484    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:51.294649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:51.294616    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:53.206962 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:53.206908    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:55.151658 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:55.151622    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:57.158983 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.158938    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:57.777063 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.776969    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"]
Oct 29 23:20:57.777742 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.777727    5428 cpu_manager.go:414] "containerMap: removing container" podUID="1b154cd7-0f58-4506-b7ac-2eed207e4a0b" containerName="istio-proxy"
Oct 29 23:20:57.777826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.777743    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b154cd7-0f58-4506-b7ac-2eed207e4a0b" containerName="istio-proxy"
Oct 29 23:20:57.777826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.777758    5428 cpu_manager.go:414] "containerMap: removing container" podUID="db52cef6-d02d-439a-97ee-685acbad592d" containerName="istio-proxy"
Oct 29 23:20:57.777826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.777766    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db52cef6-d02d-439a-97ee-685acbad592d" containerName="istio-proxy"
Oct 29 23:20:57.777933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.777838    5428 cpu_manager.go:414] "containerMap: removing container" podUID="a0b7e549-f379-4461-ad77-c3ce9c1d00fc" containerName="maas-api"
Oct 29 23:20:57.777933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.777848    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b7e549-f379-4461-ad77-c3ce9c1d00fc" containerName="maas-api"
Oct 29 23:20:57.777933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.777860    5428 cpu_manager.go:414] "containerMap: removing container" podUID="0a8f9e56-8bb0-4657-9539-f7b3aa80af15" containerName="istio-proxy"
Oct 29 23:20:57.777933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.777867    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a8f9e56-8bb0-4657-9539-f7b3aa80af15" containerName="istio-proxy"
Oct 29 23:20:57.777933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.777880    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b42c20e6-b20d-4380-a772-b06a9b3d3592" containerName="payload-pre-processing"
Oct 29 23:20:57.777933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.777887    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42c20e6-b20d-4380-a772-b06a9b3d3592" containerName="payload-pre-processing"
Oct 29 23:20:57.777933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.777898    5428 cpu_manager.go:414] "containerMap: removing container" podUID="db877578-1ef2-4a99-b585-0ea0504a9613" containerName="payload-processing"
Oct 29 23:20:57.777933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.777906    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db877578-1ef2-4a99-b585-0ea0504a9613" containerName="payload-processing"
Oct 29 23:20:57.777933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.777918    5428 cpu_manager.go:414] "containerMap: removing container" podUID="bf52a0e8-c739-47f8-9676-c5261e8d0f59" containerName="istio-proxy"
Oct 29 23:20:57.777933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.777925    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf52a0e8-c739-47f8-9676-c5261e8d0f59" containerName="istio-proxy"
Oct 29 23:20:57.778314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.778111    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b154cd7-0f58-4506-b7ac-2eed207e4a0b" containerName="istio-proxy"
Oct 29 23:20:57.778314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.778121    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="db52cef6-d02d-439a-97ee-685acbad592d" containerName="istio-proxy"
Oct 29 23:20:57.778314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.778137    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b7e549-f379-4461-ad77-c3ce9c1d00fc" containerName="maas-api"
Oct 29 23:20:57.778314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.778143    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a8f9e56-8bb0-4657-9539-f7b3aa80af15" containerName="istio-proxy"
Oct 29 23:20:57.778314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.778150    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42c20e6-b20d-4380-a772-b06a9b3d3592" containerName="payload-pre-processing"
Oct 29 23:20:57.778314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.778156    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="db877578-1ef2-4a99-b585-0ea0504a9613" containerName="payload-processing"
Oct 29 23:20:57.778314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.778162    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf52a0e8-c739-47f8-9676-c5261e8d0f59" containerName="istio-proxy"
Oct 29 23:20:57.782279 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.782251    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:57.807548 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.807517    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"]
Oct 29 23:20:57.875499 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.875464    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-7a3c1a-openshift-default-dockercfg-69xwl\""
Oct 29 23:20:57.960341 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.960307    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-workload-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:57.960535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.960363    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-data\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:57.960535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.960519    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istiod-ca-cert\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:57.960699 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.960548    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-credential-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:57.960747 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.960719    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-podinfo\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:57.960886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.960866    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-workload-certs\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:57.961007 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.960989    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-token\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:57.961064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.961039    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9hjk\" (UniqueName: \"kubernetes.io/projected/0de8513d-c8dc-4856-b5d1-8174e0676e1b-kube-api-access-f9hjk\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:57.961105 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:57.961092    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-envoy\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.062627 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.062526    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-workload-certs\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.062856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.062645    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-token\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.062856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.062715    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9hjk\" (UniqueName: \"kubernetes.io/projected/0de8513d-c8dc-4856-b5d1-8174e0676e1b-kube-api-access-f9hjk\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.062856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.062759    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-envoy\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.062856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.062813    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-workload-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.063107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.062858    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-data\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.063107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.062969    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istiod-ca-cert\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.063107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.063001    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-credential-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.063107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.063017    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-workload-certs\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.063330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.063120    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-podinfo\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.063330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.063190    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-workload-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.063330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.063271    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-data\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.063505 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.063370    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-credential-socket\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.063673 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.063648    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istiod-ca-cert\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.065195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.065169    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-envoy\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.065627 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.065610    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-podinfo\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.085728 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.085700    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-token\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.085981 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.085958    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9hjk\" (UniqueName: \"kubernetes.io/projected/0de8513d-c8dc-4856-b5d1-8174e0676e1b-kube-api-access-f9hjk\") pod \"e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") " pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.100179 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.100146    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:20:58.424851 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.424815    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"]
Oct 29 23:20:58.425364 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:20:58.425339    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de8513d_c8dc_4856_b5d1_8174e0676e1b.slice/crio-481da94aa4bd4aa2eb4db0753cfb770b8a6176b2fa793940fcd93afceaeb603f WatchSource:0}: Error finding container 481da94aa4bd4aa2eb4db0753cfb770b8a6176b2fa793940fcd93afceaeb603f: Status 404 returned error can't find the container with id 481da94aa4bd4aa2eb4db0753cfb770b8a6176b2fa793940fcd93afceaeb603f
Oct 29 23:20:58.428178 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.428141    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:20:58.428304 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.428224    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:20:58.428304 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.428270    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:20:58.916446 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:58.916386    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:20:59.257592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:59.257497    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r" event={"ID":"0de8513d-c8dc-4856-b5d1-8174e0676e1b","Type":"ContainerStarted","Data":"ea142ddd5f0bf2eda39f5e41301946b2c96333bf2318b98df651191b9aada4e3"}
Oct 29 23:20:59.257592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:59.257562    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r" event={"ID":"0de8513d-c8dc-4856-b5d1-8174e0676e1b","Type":"ContainerStarted","Data":"481da94aa4bd4aa2eb4db0753cfb770b8a6176b2fa793940fcd93afceaeb603f"}
Oct 29 23:20:59.321096 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:20:59.321035    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r" podStartSLOduration=2.321022085 podStartE2EDuration="2.321022085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:59.316506299 +0000 UTC m=+2262.659455055" watchObservedRunningTime="2026-10-29 23:20:59.321022085 +0000 UTC m=+2262.663970829"
Oct 29 23:21:00.101297 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:00.101261    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:21:00.103477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:00.103448    5428 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 29 23:21:00.103612 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:00.103497    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r" podUID="0de8513d-c8dc-4856-b5d1-8174e0676e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 29 23:21:00.840245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:00.840202    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:01.100683 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:01.100603    5428 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 29 23:21:01.100683 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:01.100663    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r" podUID="0de8513d-c8dc-4856-b5d1-8174e0676e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 29 23:21:02.101084 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:02.101045    5428 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 29 23:21:02.101467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:02.101112    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r" podUID="0de8513d-c8dc-4856-b5d1-8174e0676e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 29 23:21:02.393571 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:02.392997    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"]
Oct 29 23:21:02.393571 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:02.393471    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv" podUID="dadecf77-4bd8-4ca8-bd71-3b2f36b62953" containerName="maas-api" containerID="cri-o://506fcd29817cf986e434dcd91dc11f434476b62936da5f71ce2c5f8c03f312dc" gracePeriod=30
Oct 29 23:21:02.442686 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:02.442650    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:02.502317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:02.502275    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv" podUID="dadecf77-4bd8-4ca8-bd71-3b2f36b62953" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.198:8443/health\": dial tcp 10.133.0.198:8443: connect: connection refused"
Oct 29 23:21:02.984025 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:02.983989    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"
Oct 29 23:21:03.037408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.037375    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-openshift-service-ca\") pod \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\" (UID: \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\") "
Oct 29 23:21:03.037703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.037682    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-openshift-service-ca" pod "dadecf77-4bd8-4ca8-bd71-3b2f36b62953" (UID: "dadecf77-4bd8-4ca8-bd71-3b2f36b62953"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:21:03.037894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.037749    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-metrics-certs\") pod \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\" (UID: \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\") "
Oct 29 23:21:03.038170 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.038149    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-kube-api-access-b47z6\" (UniqueName: \"kubernetes.io/projected/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-kube-api-access-b47z6\") pod \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\" (UID: \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\") "
Oct 29 23:21:03.038887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.038864    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-maas-api-tls\") pod \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\" (UID: \"dadecf77-4bd8-4ca8-bd71-3b2f36b62953\") "
Oct 29 23:21:03.040033 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.039988    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:03.040368 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.040333    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-kube-api-access-b47z6" pod "dadecf77-4bd8-4ca8-bd71-3b2f36b62953" (UID: "dadecf77-4bd8-4ca8-bd71-3b2f36b62953"). InnerVolumeSpecName "kube-api-access-b47z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:03.040911 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.040885    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-metrics-certs" pod "dadecf77-4bd8-4ca8-bd71-3b2f36b62953" (UID: "dadecf77-4bd8-4ca8-bd71-3b2f36b62953"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:21:03.050999 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.048509    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-maas-api-tls" pod "dadecf77-4bd8-4ca8-bd71-3b2f36b62953" (UID: "dadecf77-4bd8-4ca8-bd71-3b2f36b62953"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:21:03.103831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.102928    5428 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 29 23:21:03.103831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.102986    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r" podUID="0de8513d-c8dc-4856-b5d1-8174e0676e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 29 23:21:03.142215 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.142174    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:03.142377 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.142222    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b47z6\" (UniqueName: \"kubernetes.io/projected/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-kube-api-access-b47z6\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:03.142377 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.142239    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dadecf77-4bd8-4ca8-bd71-3b2f36b62953-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:03.251005 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.250902    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:03.317843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.317734    5428 generic.go:356] "Generic (PLEG): container finished" podID="dadecf77-4bd8-4ca8-bd71-3b2f36b62953" containerID="506fcd29817cf986e434dcd91dc11f434476b62936da5f71ce2c5f8c03f312dc" exitCode=0
Oct 29 23:21:03.317843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.317778    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"
Oct 29 23:21:03.318058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.317910    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv" event={"ID":"dadecf77-4bd8-4ca8-bd71-3b2f36b62953","Type":"ContainerDied","Data":"506fcd29817cf986e434dcd91dc11f434476b62936da5f71ce2c5f8c03f312dc"}
Oct 29 23:21:03.318058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.317970    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv" event={"ID":"dadecf77-4bd8-4ca8-bd71-3b2f36b62953","Type":"ContainerDied","Data":"6ea967f060f6381a9ef7b7fa956cf6bba16e5df05844f4baa777ccbe707ef238"}
Oct 29 23:21:03.318058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.317998    5428 scope.go:122] "RemoveContainer" containerID="506fcd29817cf986e434dcd91dc11f434476b62936da5f71ce2c5f8c03f312dc"
Oct 29 23:21:03.334810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.334778    5428 scope.go:122] "RemoveContainer" containerID="506fcd29817cf986e434dcd91dc11f434476b62936da5f71ce2c5f8c03f312dc"
Oct 29 23:21:03.335067 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:21:03.335047    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"506fcd29817cf986e434dcd91dc11f434476b62936da5f71ce2c5f8c03f312dc\": container with ID starting with 506fcd29817cf986e434dcd91dc11f434476b62936da5f71ce2c5f8c03f312dc not found: ID does not exist" containerID="506fcd29817cf986e434dcd91dc11f434476b62936da5f71ce2c5f8c03f312dc"
Oct 29 23:21:03.335128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.335074    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506fcd29817cf986e434dcd91dc11f434476b62936da5f71ce2c5f8c03f312dc"} err="failed to get container status \"506fcd29817cf986e434dcd91dc11f434476b62936da5f71ce2c5f8c03f312dc\": rpc error: code = NotFound desc = could not find container \"506fcd29817cf986e434dcd91dc11f434476b62936da5f71ce2c5f8c03f312dc\": container with ID starting with 506fcd29817cf986e434dcd91dc11f434476b62936da5f71ce2c5f8c03f312dc not found: ID does not exist"
Oct 29 23:21:03.389081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.389041    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"]
Oct 29 23:21:03.393502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.393476    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a3c1a-8db8b5695-vcqnv"]
Oct 29 23:21:03.563824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.559113    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:03.693167 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:03.693109    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:04.083917 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:04.083621    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:04.101093 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:04.101057    5428 patch_prober.go:29] interesting pod/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 29 23:21:04.101260 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:04.101125    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r" podUID="0de8513d-c8dc-4856-b5d1-8174e0676e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 29 23:21:04.781840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:04.781756    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dadecf77-4bd8-4ca8-bd71-3b2f36b62953" path="/var/lib/kubelet/pods/dadecf77-4bd8-4ca8-bd71-3b2f36b62953/volumes"
Oct 29 23:21:05.105757 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:05.105671    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:21:05.106184 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:05.106171    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:21:05.106996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:05.106962    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:21:05.854713 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:05.854679    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:06.631019 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:06.630986    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:07.687546 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:07.687516    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:07.961667 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:07.961563    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:08.432488 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:08.432455    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:08.545761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:08.545729    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:09.917750 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:09.917720    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:10.352988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:10.352908    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:10.477696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:10.477662    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:10.658833 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:10.658631    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"]
Oct 29 23:21:10.659028 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:10.658912    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r" podUID="0de8513d-c8dc-4856-b5d1-8174e0676e1b" containerName="istio-proxy" containerID="cri-o://ea142ddd5f0bf2eda39f5e41301946b2c96333bf2318b98df651191b9aada4e3" gracePeriod=30
Oct 29 23:21:10.718186 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:10.718144    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:11.730624 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:11.730585    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:11.853950 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:11.853890    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:12.088831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:12.088739    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:13.387568 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:13.387536    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:13.516149 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:13.516119    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:14.221878 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:14.221847    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:15.956794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:15.956764    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:21:16.017505 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.017466    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istiod-ca-cert\") pod \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") "
Oct 29 23:21:16.017826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.017782    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-token\" (UniqueName: \"kubernetes.io/projected/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-token\") pod \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") "
Oct 29 23:21:16.018067 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.018040    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-data\") pod \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") "
Oct 29 23:21:16.018223 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.018126    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-workload-socket\") pod \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") "
Oct 29 23:21:16.018223 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.018164    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:16.018223 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.018216    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de8513d-c8dc-4856-b5d1-8174e0676e1b-kube-api-access-f9hjk\" (UniqueName: \"kubernetes.io/projected/0de8513d-c8dc-4856-b5d1-8174e0676e1b-kube-api-access-f9hjk\") pod \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") "
Oct 29 23:21:16.018479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.018258    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-workload-certs\") pod \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") "
Oct 29 23:21:16.018479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.018302    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-envoy\") pod \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") "
Oct 29 23:21:16.018479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.018338    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-credential-socket\") pod \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") "
Oct 29 23:21:16.018867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.018518    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-podinfo\") pod \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\" (UID: \"0de8513d-c8dc-4856-b5d1-8174e0676e1b\") "
Oct 29 23:21:16.018867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.017919    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istiod-ca-cert" pod "0de8513d-c8dc-4856-b5d1-8174e0676e1b" (UID: "0de8513d-c8dc-4856-b5d1-8174e0676e1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:21:16.018867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.018562    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-data" pod "0de8513d-c8dc-4856-b5d1-8174e0676e1b" (UID: "0de8513d-c8dc-4856-b5d1-8174e0676e1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:16.018867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.018609    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-workload-certs" pod "0de8513d-c8dc-4856-b5d1-8174e0676e1b" (UID: "0de8513d-c8dc-4856-b5d1-8174e0676e1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:16.018867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.018785    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-workload-socket" pod "0de8513d-c8dc-4856-b5d1-8174e0676e1b" (UID: "0de8513d-c8dc-4856-b5d1-8174e0676e1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:16.019554 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.019112    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-credential-socket" pod "0de8513d-c8dc-4856-b5d1-8174e0676e1b" (UID: "0de8513d-c8dc-4856-b5d1-8174e0676e1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:16.019554 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.019310    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:16.019554 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.019327    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:16.019554 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.019341    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:16.019554 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.019353    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:16.019554 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.019367    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:16.021409 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.021377    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-podinfo" pod "0de8513d-c8dc-4856-b5d1-8174e0676e1b" (UID: "0de8513d-c8dc-4856-b5d1-8174e0676e1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:21:16.022408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.022381    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-token" pod "0de8513d-c8dc-4856-b5d1-8174e0676e1b" (UID: "0de8513d-c8dc-4856-b5d1-8174e0676e1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:16.023670 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.023639    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-envoy" pod "0de8513d-c8dc-4856-b5d1-8174e0676e1b" (UID: "0de8513d-c8dc-4856-b5d1-8174e0676e1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:16.023828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.023788    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de8513d-c8dc-4856-b5d1-8174e0676e1b-kube-api-access-f9hjk" pod "0de8513d-c8dc-4856-b5d1-8174e0676e1b" (UID: "0de8513d-c8dc-4856-b5d1-8174e0676e1b"). InnerVolumeSpecName "kube-api-access-f9hjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:16.128156 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.128119    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:16.128156 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.128153    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:16.128442 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.128167    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9hjk\" (UniqueName: \"kubernetes.io/projected/0de8513d-c8dc-4856-b5d1-8174e0676e1b-kube-api-access-f9hjk\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:16.128442 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.128182    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de8513d-c8dc-4856-b5d1-8174e0676e1b-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:16.426051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.426018    5428 generic.go:356] "Generic (PLEG): container finished" podID="0de8513d-c8dc-4856-b5d1-8174e0676e1b" containerID="ea142ddd5f0bf2eda39f5e41301946b2c96333bf2318b98df651191b9aada4e3" exitCode=0
Oct 29 23:21:16.426241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.426068    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"
Oct 29 23:21:16.426241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.426097    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r" event={"ID":"0de8513d-c8dc-4856-b5d1-8174e0676e1b","Type":"ContainerDied","Data":"ea142ddd5f0bf2eda39f5e41301946b2c96333bf2318b98df651191b9aada4e3"}
Oct 29 23:21:16.426241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.426157    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r" event={"ID":"0de8513d-c8dc-4856-b5d1-8174e0676e1b","Type":"ContainerDied","Data":"481da94aa4bd4aa2eb4db0753cfb770b8a6176b2fa793940fcd93afceaeb603f"}
Oct 29 23:21:16.426241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.426180    5428 scope.go:122] "RemoveContainer" containerID="ea142ddd5f0bf2eda39f5e41301946b2c96333bf2318b98df651191b9aada4e3"
Oct 29 23:21:16.438919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.438901    5428 scope.go:122] "RemoveContainer" containerID="ea142ddd5f0bf2eda39f5e41301946b2c96333bf2318b98df651191b9aada4e3"
Oct 29 23:21:16.439188 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:21:16.439169    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea142ddd5f0bf2eda39f5e41301946b2c96333bf2318b98df651191b9aada4e3\": container with ID starting with ea142ddd5f0bf2eda39f5e41301946b2c96333bf2318b98df651191b9aada4e3 not found: ID does not exist" containerID="ea142ddd5f0bf2eda39f5e41301946b2c96333bf2318b98df651191b9aada4e3"
Oct 29 23:21:16.439242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.439196    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea142ddd5f0bf2eda39f5e41301946b2c96333bf2318b98df651191b9aada4e3"} err="failed to get container status \"ea142ddd5f0bf2eda39f5e41301946b2c96333bf2318b98df651191b9aada4e3\": rpc error: code = NotFound desc = could not find container \"ea142ddd5f0bf2eda39f5e41301946b2c96333bf2318b98df651191b9aada4e3\": container with ID starting with ea142ddd5f0bf2eda39f5e41301946b2c96333bf2318b98df651191b9aada4e3 not found: ID does not exist"
Oct 29 23:21:16.481383 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.481347    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"]
Oct 29 23:21:16.493144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.493096    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a3c1a-openshift-default-785c8cb669-htm8r"]
Oct 29 23:21:16.778745 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:16.778673    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0de8513d-c8dc-4856-b5d1-8174e0676e1b" path="/var/lib/kubelet/pods/0de8513d-c8dc-4856-b5d1-8174e0676e1b/volumes"
Oct 29 23:21:17.369656 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:21:17.369621    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:21:17.840724 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:17.840644    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:19.549740 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:19.549709    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:20.100427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:20.100393    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"]
Oct 29 23:21:20.100941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:20.100906    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4" podUID="76bcb1e7-3c9e-4222-abc4-96c5c461707c" containerName="istio-proxy" containerID="cri-o://d19f9adf5eac8ecc02df18d38a90ff407b917ad2dc692d7530f63a313b95fc43" gracePeriod=30
Oct 29 23:21:21.399300 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:21.399243    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:23.125021 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:23.124985    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:24.903069 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:24.903038    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:25.396379 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.396348    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:21:25.476976 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.476893    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-token\" (UniqueName: \"kubernetes.io/projected/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-token\") pod \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") "
Oct 29 23:21:25.477206 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.477186    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-credential-socket\") pod \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") "
Oct 29 23:21:25.477370 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.477356    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-envoy\") pod \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") "
Oct 29 23:21:25.478086 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.477575    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-credential-socket" pod "76bcb1e7-3c9e-4222-abc4-96c5c461707c" (UID: "76bcb1e7-3c9e-4222-abc4-96c5c461707c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:25.478677 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.478440    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-workload-certs" pod "76bcb1e7-3c9e-4222-abc4-96c5c461707c" (UID: "76bcb1e7-3c9e-4222-abc4-96c5c461707c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:25.478914 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.478844    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-workload-certs\") pod \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") "
Oct 29 23:21:25.479080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.479065    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-data\") pod \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") "
Oct 29 23:21:25.479433 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.479418    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istiod-ca-cert\") pod \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") "
Oct 29 23:21:25.479943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.479925    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-workload-socket\") pod \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") "
Oct 29 23:21:25.480468 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.479339    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-data" pod "76bcb1e7-3c9e-4222-abc4-96c5c461707c" (UID: "76bcb1e7-3c9e-4222-abc4-96c5c461707c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:25.480593 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.479864    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istiod-ca-cert" pod "76bcb1e7-3c9e-4222-abc4-96c5c461707c" (UID: "76bcb1e7-3c9e-4222-abc4-96c5c461707c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:21:25.480593 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.480233    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-token" pod "76bcb1e7-3c9e-4222-abc4-96c5c461707c" (UID: "76bcb1e7-3c9e-4222-abc4-96c5c461707c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:25.480593 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.480293    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-workload-socket" pod "76bcb1e7-3c9e-4222-abc4-96c5c461707c" (UID: "76bcb1e7-3c9e-4222-abc4-96c5c461707c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:25.480593 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.480351    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-envoy" pod "76bcb1e7-3c9e-4222-abc4-96c5c461707c" (UID: "76bcb1e7-3c9e-4222-abc4-96c5c461707c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:25.481111 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.481089    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:25.481195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.481116    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:25.481195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.481130    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:25.481195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.481143    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:25.481195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.481155    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:25.481195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.481170    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:25.481195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.481183    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:25.508990 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.508954    5428 generic.go:356] "Generic (PLEG): container finished" podID="76bcb1e7-3c9e-4222-abc4-96c5c461707c" containerID="d19f9adf5eac8ecc02df18d38a90ff407b917ad2dc692d7530f63a313b95fc43" exitCode=0
Oct 29 23:21:25.509121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.509031    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"
Oct 29 23:21:25.509186 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.509122    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4" event={"ID":"76bcb1e7-3c9e-4222-abc4-96c5c461707c","Type":"ContainerDied","Data":"d19f9adf5eac8ecc02df18d38a90ff407b917ad2dc692d7530f63a313b95fc43"}
Oct 29 23:21:25.509186 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.509175    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4" event={"ID":"76bcb1e7-3c9e-4222-abc4-96c5c461707c","Type":"ContainerDied","Data":"e731972e429b4d80c3b992ded9fdf15bffb5ff6f6dc34b2ade3edf7066ace588"}
Oct 29 23:21:25.509309 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.509205    5428 scope.go:122] "RemoveContainer" containerID="d19f9adf5eac8ecc02df18d38a90ff407b917ad2dc692d7530f63a313b95fc43"
Oct 29 23:21:25.523643 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.523616    5428 scope.go:122] "RemoveContainer" containerID="d19f9adf5eac8ecc02df18d38a90ff407b917ad2dc692d7530f63a313b95fc43"
Oct 29 23:21:25.523973 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:21:25.523950    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19f9adf5eac8ecc02df18d38a90ff407b917ad2dc692d7530f63a313b95fc43\": container with ID starting with d19f9adf5eac8ecc02df18d38a90ff407b917ad2dc692d7530f63a313b95fc43 not found: ID does not exist" containerID="d19f9adf5eac8ecc02df18d38a90ff407b917ad2dc692d7530f63a313b95fc43"
Oct 29 23:21:25.524065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.523986    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19f9adf5eac8ecc02df18d38a90ff407b917ad2dc692d7530f63a313b95fc43"} err="failed to get container status \"d19f9adf5eac8ecc02df18d38a90ff407b917ad2dc692d7530f63a313b95fc43\": rpc error: code = NotFound desc = could not find container \"d19f9adf5eac8ecc02df18d38a90ff407b917ad2dc692d7530f63a313b95fc43\": container with ID starting with d19f9adf5eac8ecc02df18d38a90ff407b917ad2dc692d7530f63a313b95fc43 not found: ID does not exist"
Oct 29 23:21:25.582424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.582393    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bcb1e7-3c9e-4222-abc4-96c5c461707c-kube-api-access-n6rwk\" (UniqueName: \"kubernetes.io/projected/76bcb1e7-3c9e-4222-abc4-96c5c461707c-kube-api-access-n6rwk\") pod \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") "
Oct 29 23:21:25.582598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.582580    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-podinfo\") pod \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\" (UID: \"76bcb1e7-3c9e-4222-abc4-96c5c461707c\") "
Oct 29 23:21:25.584612 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.584587    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bcb1e7-3c9e-4222-abc4-96c5c461707c-kube-api-access-n6rwk" pod "76bcb1e7-3c9e-4222-abc4-96c5c461707c" (UID: "76bcb1e7-3c9e-4222-abc4-96c5c461707c"). InnerVolumeSpecName "kube-api-access-n6rwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:25.584728 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.584583    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-podinfo" pod "76bcb1e7-3c9e-4222-abc4-96c5c461707c" (UID: "76bcb1e7-3c9e-4222-abc4-96c5c461707c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:21:25.684390 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.684359    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6rwk\" (UniqueName: \"kubernetes.io/projected/76bcb1e7-3c9e-4222-abc4-96c5c461707c-kube-api-access-n6rwk\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:25.684390 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.684388    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bcb1e7-3c9e-4222-abc4-96c5c461707c-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:25.859191 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.859156    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"]
Oct 29 23:21:25.871170 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:25.871130    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-dkfk4"]
Oct 29 23:21:26.778624 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:26.778593    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76bcb1e7-3c9e-4222-abc4-96c5c461707c" path="/var/lib/kubelet/pods/76bcb1e7-3c9e-4222-abc4-96c5c461707c/volumes"
Oct 29 23:21:26.807311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:26.807283    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:28.510699 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:28.510666    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:29.177460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:29.177431    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:30.378110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:30.378069    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:31.996858 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:31.996821    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:32.267795 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:32.267683    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:32.554155 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:32.554062    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:32.980128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:32.980094    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:34.463988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:34.463952    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:37.241765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:37.241730    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:37.394372 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:37.394336    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:37.517237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:37.517126    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:38.997217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:38.997178    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:39.133247 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:39.133167    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:39.144115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:39.144088    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"]
Oct 29 23:21:39.144307 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:39.144281    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj" podUID="8e3f78e3-9804-4b4f-8e9d-6e37b0351643" containerName="istio-proxy" containerID="cri-o://053af7b4509390659618ec8cc86d2bdc37944940469e3a1574727290435d520c" gracePeriod=30
Oct 29 23:21:40.134733 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:40.134694    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:42.299178 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:42.299138    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:42.943071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:42.943037    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"]
Oct 29 23:21:42.943328 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:42.943296    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh" podUID="2eef2e22-9704-4291-b7ae-91c29acd11de" containerName="istio-proxy" containerID="cri-o://b1f9dff7048aec5eb8ca721e37f148e3fb07db50bab779d76ff357aa43c3bede" gracePeriod=30
Oct 29 23:21:44.187736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.187698    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:44.449426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.449402    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:21:44.535042 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.535005    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-token\" (UniqueName: \"kubernetes.io/projected/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-token\") pod \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") "
Oct 29 23:21:44.535240 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.535121    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-podinfo\") pod \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") "
Oct 29 23:21:44.535240 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.535191    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-workload-socket\") pod \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") "
Oct 29 23:21:44.535240 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.535227    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-envoy\") pod \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") "
Oct 29 23:21:44.535436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.535304    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-data\") pod \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") "
Oct 29 23:21:44.535436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.535370    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-kube-api-access-jsw76\" (UniqueName: \"kubernetes.io/projected/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-kube-api-access-jsw76\") pod \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") "
Oct 29 23:21:44.535436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.535411    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istiod-ca-cert\") pod \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") "
Oct 29 23:21:44.535612 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.535546    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-credential-socket\") pod \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") "
Oct 29 23:21:44.535672 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.535605    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-data" pod "8e3f78e3-9804-4b4f-8e9d-6e37b0351643" (UID: "8e3f78e3-9804-4b4f-8e9d-6e37b0351643"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:44.535672 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.535623    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-workload-certs\") pod \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\" (UID: \"8e3f78e3-9804-4b4f-8e9d-6e37b0351643\") "
Oct 29 23:21:44.535858 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.535834    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-credential-socket" pod "8e3f78e3-9804-4b4f-8e9d-6e37b0351643" (UID: "8e3f78e3-9804-4b4f-8e9d-6e37b0351643"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:44.536029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.535868    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istiod-ca-cert" pod "8e3f78e3-9804-4b4f-8e9d-6e37b0351643" (UID: "8e3f78e3-9804-4b4f-8e9d-6e37b0351643"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:21:44.536029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.535883    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-workload-socket" pod "8e3f78e3-9804-4b4f-8e9d-6e37b0351643" (UID: "8e3f78e3-9804-4b4f-8e9d-6e37b0351643"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:44.536235 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.536188    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-workload-certs" pod "8e3f78e3-9804-4b4f-8e9d-6e37b0351643" (UID: "8e3f78e3-9804-4b4f-8e9d-6e37b0351643"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:44.536717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.536692    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:44.536915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.536891    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:44.537025 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.536918    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:44.537025 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.536934    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:44.537025 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.536950    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:44.537939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.537913    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-podinfo" pod "8e3f78e3-9804-4b4f-8e9d-6e37b0351643" (UID: "8e3f78e3-9804-4b4f-8e9d-6e37b0351643"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:21:44.538335 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.538301    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-envoy" pod "8e3f78e3-9804-4b4f-8e9d-6e37b0351643" (UID: "8e3f78e3-9804-4b4f-8e9d-6e37b0351643"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:44.538335 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.538321    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-kube-api-access-jsw76" pod "8e3f78e3-9804-4b4f-8e9d-6e37b0351643" (UID: "8e3f78e3-9804-4b4f-8e9d-6e37b0351643"). InnerVolumeSpecName "kube-api-access-jsw76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:44.538927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.538903    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-token" pod "8e3f78e3-9804-4b4f-8e9d-6e37b0351643" (UID: "8e3f78e3-9804-4b4f-8e9d-6e37b0351643"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:44.638603 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.638565    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsw76\" (UniqueName: \"kubernetes.io/projected/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-kube-api-access-jsw76\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:44.638603 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.638603    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:44.638827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.638618    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:44.638827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.638631    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3f78e3-9804-4b4f-8e9d-6e37b0351643-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:44.683246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.683152    5428 generic.go:356] "Generic (PLEG): container finished" podID="8e3f78e3-9804-4b4f-8e9d-6e37b0351643" containerID="053af7b4509390659618ec8cc86d2bdc37944940469e3a1574727290435d520c" exitCode=0
Oct 29 23:21:44.683246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.683194    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj" event={"ID":"8e3f78e3-9804-4b4f-8e9d-6e37b0351643","Type":"ContainerDied","Data":"053af7b4509390659618ec8cc86d2bdc37944940469e3a1574727290435d520c"}
Oct 29 23:21:44.683246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.683213    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"
Oct 29 23:21:44.683576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.683255    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj" event={"ID":"8e3f78e3-9804-4b4f-8e9d-6e37b0351643","Type":"ContainerDied","Data":"62c6b0743e3c6168b7e78ae66104885a239fbea1d5356a12850e047dd1043db7"}
Oct 29 23:21:44.683576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.683283    5428 scope.go:122] "RemoveContainer" containerID="053af7b4509390659618ec8cc86d2bdc37944940469e3a1574727290435d520c"
Oct 29 23:21:44.696710 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.696692    5428 scope.go:122] "RemoveContainer" containerID="053af7b4509390659618ec8cc86d2bdc37944940469e3a1574727290435d520c"
Oct 29 23:21:44.697006 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:21:44.696982    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"053af7b4509390659618ec8cc86d2bdc37944940469e3a1574727290435d520c\": container with ID starting with 053af7b4509390659618ec8cc86d2bdc37944940469e3a1574727290435d520c not found: ID does not exist" containerID="053af7b4509390659618ec8cc86d2bdc37944940469e3a1574727290435d520c"
Oct 29 23:21:44.697117 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.697014    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053af7b4509390659618ec8cc86d2bdc37944940469e3a1574727290435d520c"} err="failed to get container status \"053af7b4509390659618ec8cc86d2bdc37944940469e3a1574727290435d520c\": rpc error: code = NotFound desc = could not find container \"053af7b4509390659618ec8cc86d2bdc37944940469e3a1574727290435d520c\": container with ID starting with 053af7b4509390659618ec8cc86d2bdc37944940469e3a1574727290435d520c not found: ID does not exist"
Oct 29 23:21:44.738412 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.738384    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"]
Oct 29 23:21:44.743218 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.743194    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-mrqrj"]
Oct 29 23:21:44.778511 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:44.778485    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e3f78e3-9804-4b4f-8e9d-6e37b0351643" path="/var/lib/kubelet/pods/8e3f78e3-9804-4b4f-8e9d-6e37b0351643/volumes"
Oct 29 23:21:46.029736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:46.029680    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:47.371043 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:21:47.371002    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:21:47.889163 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:47.889130    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:48.017298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.017267    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"]
Oct 29 23:21:48.017663 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.017635    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c" podUID="7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb" containerName="discovery" containerID="cri-o://3d60f1706b8bafeb32527bdeaf2daa9a5edf877c16309c83e8fedcb40f8d88e8" gracePeriod=30
Oct 29 23:21:48.086412 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.086386    5428 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vs95c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.114:8080/ready\": dial tcp 10.133.0.114:8080: connect: connection refused" start-of-body=
Oct 29 23:21:48.086542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.086441    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c" podUID="7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.114:8080/ready\": dial tcp 10.133.0.114:8080: connect: connection refused"
Oct 29 23:21:48.323090 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.323062    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:21:48.361989 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.361964    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 23:21:48.396000 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.395916    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-token\" (UniqueName: \"kubernetes.io/projected/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-token\") pod \"2eef2e22-9704-4291-b7ae-91c29acd11de\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") "
Oct 29 23:21:48.397094 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.396694    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-envoy\") pod \"2eef2e22-9704-4291-b7ae-91c29acd11de\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") "
Oct 29 23:21:48.397391 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.397372    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-credential-socket\") pod \"2eef2e22-9704-4291-b7ae-91c29acd11de\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") "
Oct 29 23:21:48.397699 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.397677    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-workload-socket\") pod \"2eef2e22-9704-4291-b7ae-91c29acd11de\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") "
Oct 29 23:21:48.397792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.397744    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-credential-socket" pod "2eef2e22-9704-4291-b7ae-91c29acd11de" (UID: "2eef2e22-9704-4291-b7ae-91c29acd11de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:48.397869 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.397828    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-workload-certs\") pod \"2eef2e22-9704-4291-b7ae-91c29acd11de\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") "
Oct 29 23:21:48.397942 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.397923    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-workload-socket" pod "2eef2e22-9704-4291-b7ae-91c29acd11de" (UID: "2eef2e22-9704-4291-b7ae-91c29acd11de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:48.398004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.397949    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-podinfo\") pod \"2eef2e22-9704-4291-b7ae-91c29acd11de\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") "
Oct 29 23:21:48.398093 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.398079    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-data\") pod \"2eef2e22-9704-4291-b7ae-91c29acd11de\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") "
Oct 29 23:21:48.398168 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.398108    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-workload-certs" pod "2eef2e22-9704-4291-b7ae-91c29acd11de" (UID: "2eef2e22-9704-4291-b7ae-91c29acd11de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:48.398168 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.398141    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eef2e22-9704-4291-b7ae-91c29acd11de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eef2e22-9704-4291-b7ae-91c29acd11de-istiod-ca-cert\") pod \"2eef2e22-9704-4291-b7ae-91c29acd11de\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") "
Oct 29 23:21:48.398290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.398184    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eef2e22-9704-4291-b7ae-91c29acd11de-kube-api-access-9v6wm\" (UniqueName: \"kubernetes.io/projected/2eef2e22-9704-4291-b7ae-91c29acd11de-kube-api-access-9v6wm\") pod \"2eef2e22-9704-4291-b7ae-91c29acd11de\" (UID: \"2eef2e22-9704-4291-b7ae-91c29acd11de\") "
Oct 29 23:21:48.398504 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.398401    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-data" pod "2eef2e22-9704-4291-b7ae-91c29acd11de" (UID: "2eef2e22-9704-4291-b7ae-91c29acd11de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:48.398854 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.398828    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eef2e22-9704-4291-b7ae-91c29acd11de-istiod-ca-cert" pod "2eef2e22-9704-4291-b7ae-91c29acd11de" (UID: "2eef2e22-9704-4291-b7ae-91c29acd11de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:21:48.398854 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.398836    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.399043 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.398867    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.399043 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.398878    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.399043 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.398887    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.403252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.403227    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-envoy" pod "2eef2e22-9704-4291-b7ae-91c29acd11de" (UID: "2eef2e22-9704-4291-b7ae-91c29acd11de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:48.406166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.406139    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-podinfo" pod "2eef2e22-9704-4291-b7ae-91c29acd11de" (UID: "2eef2e22-9704-4291-b7ae-91c29acd11de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:21:48.406166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.406147    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eef2e22-9704-4291-b7ae-91c29acd11de-kube-api-access-9v6wm" pod "2eef2e22-9704-4291-b7ae-91c29acd11de" (UID: "2eef2e22-9704-4291-b7ae-91c29acd11de"). InnerVolumeSpecName "kube-api-access-9v6wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:48.406304 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.406174    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-token" pod "2eef2e22-9704-4291-b7ae-91c29acd11de" (UID: "2eef2e22-9704-4291-b7ae-91c29acd11de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:48.499275 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.499238    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-local-certs\") pod \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") "
Oct 29 23:21:48.499503 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.499384    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-csr-dns-cert\") pod \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") "
Oct 29 23:21:48.499503 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.499427    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-cacerts\" (UniqueName: \"kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-cacerts\") pod \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") "
Oct 29 23:21:48.499648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.499511    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-csr-ca-configmap\") pod \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") "
Oct 29 23:21:48.499648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.499578    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-kube-api-access-xh4cn\" (UniqueName: \"kubernetes.io/projected/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-kube-api-access-xh4cn\") pod \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") "
Oct 29 23:21:48.499648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.499619    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-token\" (UniqueName: \"kubernetes.io/projected/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-token\") pod \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") "
Oct 29 23:21:48.499846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.499655    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-kubeconfig\") pod \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\" (UID: \"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb\") "
Oct 29 23:21:48.499975 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.499951    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-csr-ca-configmap" pod "7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb" (UID: "7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:21:48.500462 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.500432    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.500590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.500466    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.500590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.500481    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eef2e22-9704-4291-b7ae-91c29acd11de-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.500590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.500495    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v6wm\" (UniqueName: \"kubernetes.io/projected/2eef2e22-9704-4291-b7ae-91c29acd11de-kube-api-access-9v6wm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.500590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.500509    5428 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-csr-ca-configmap\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.500590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.500523    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eef2e22-9704-4291-b7ae-91c29acd11de-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.502065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.502036    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-local-certs" pod "7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb" (UID: "7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:21:48.502185 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.502081    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-cacerts" pod "7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb" (UID: "7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:21:48.502185 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.502108    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-csr-dns-cert" pod "7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb" (UID: "7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:21:48.502360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.502342    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-kube-api-access-xh4cn" pod "7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb" (UID: "7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb"). InnerVolumeSpecName "kube-api-access-xh4cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:48.502518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.502503    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-token" pod "7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb" (UID: "7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:21:48.502626 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.502614    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-kubeconfig" pod "7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb" (UID: "7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:21:48.601711 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.601675    5428 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-kubeconfig\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.601711 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.601704    5428 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-local-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.601711 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.601713    5428 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-csr-dns-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.601999 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.601723    5428 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-cacerts\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.601999 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.601733    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh4cn\" (UniqueName: \"kubernetes.io/projected/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-kube-api-access-xh4cn\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.601999 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.601742    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:21:48.716860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.716776    5428 generic.go:356] "Generic (PLEG): container finished" podID="7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb" containerID="3d60f1706b8bafeb32527bdeaf2daa9a5edf877c16309c83e8fedcb40f8d88e8" exitCode=0
Oct 29 23:21:48.716860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.716827    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c" event={"ID":"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb","Type":"ContainerDied","Data":"3d60f1706b8bafeb32527bdeaf2daa9a5edf877c16309c83e8fedcb40f8d88e8"}
Oct 29 23:21:48.717064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.716879    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"
Oct 29 23:21:48.717064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.716903    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c" event={"ID":"7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb","Type":"ContainerDied","Data":"cf2a9e7e795b3ddeb49981b8321d1bc138d4e41ddd5bd170cfc890d59fd6406a"}
Oct 29 23:21:48.717064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.716920    5428 scope.go:122] "RemoveContainer" containerID="3d60f1706b8bafeb32527bdeaf2daa9a5edf877c16309c83e8fedcb40f8d88e8"
Oct 29 23:21:48.718489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.718463    5428 generic.go:356] "Generic (PLEG): container finished" podID="2eef2e22-9704-4291-b7ae-91c29acd11de" containerID="b1f9dff7048aec5eb8ca721e37f148e3fb07db50bab779d76ff357aa43c3bede" exitCode=0
Oct 29 23:21:48.718625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.718502    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh" event={"ID":"2eef2e22-9704-4291-b7ae-91c29acd11de","Type":"ContainerDied","Data":"b1f9dff7048aec5eb8ca721e37f148e3fb07db50bab779d76ff357aa43c3bede"}
Oct 29 23:21:48.718625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.718531    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"
Oct 29 23:21:48.718625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.718547    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh" event={"ID":"2eef2e22-9704-4291-b7ae-91c29acd11de","Type":"ContainerDied","Data":"eccbc26cadcdfd99080872d26968b65106ece9eb177c4ae14b61ad030f87fbe5"}
Oct 29 23:21:48.770253 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.770196    5428 scope.go:122] "RemoveContainer" containerID="3d60f1706b8bafeb32527bdeaf2daa9a5edf877c16309c83e8fedcb40f8d88e8"
Oct 29 23:21:48.770793 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:21:48.770755    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d60f1706b8bafeb32527bdeaf2daa9a5edf877c16309c83e8fedcb40f8d88e8\": container with ID starting with 3d60f1706b8bafeb32527bdeaf2daa9a5edf877c16309c83e8fedcb40f8d88e8 not found: ID does not exist" containerID="3d60f1706b8bafeb32527bdeaf2daa9a5edf877c16309c83e8fedcb40f8d88e8"
Oct 29 23:21:48.770933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.770835    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d60f1706b8bafeb32527bdeaf2daa9a5edf877c16309c83e8fedcb40f8d88e8"} err="failed to get container status \"3d60f1706b8bafeb32527bdeaf2daa9a5edf877c16309c83e8fedcb40f8d88e8\": rpc error: code = NotFound desc = could not find container \"3d60f1706b8bafeb32527bdeaf2daa9a5edf877c16309c83e8fedcb40f8d88e8\": container with ID starting with 3d60f1706b8bafeb32527bdeaf2daa9a5edf877c16309c83e8fedcb40f8d88e8 not found: ID does not exist"
Oct 29 23:21:48.770933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.770861    5428 scope.go:122] "RemoveContainer" containerID="b1f9dff7048aec5eb8ca721e37f148e3fb07db50bab779d76ff357aa43c3bede"
Oct 29 23:21:48.782513 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.782494    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"]
Oct 29 23:21:48.782612 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.782521    5428 scope.go:122] "RemoveContainer" containerID="b1f9dff7048aec5eb8ca721e37f148e3fb07db50bab779d76ff357aa43c3bede"
Oct 29 23:21:48.782871 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:21:48.782849    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f9dff7048aec5eb8ca721e37f148e3fb07db50bab779d76ff357aa43c3bede\": container with ID starting with b1f9dff7048aec5eb8ca721e37f148e3fb07db50bab779d76ff357aa43c3bede not found: ID does not exist" containerID="b1f9dff7048aec5eb8ca721e37f148e3fb07db50bab779d76ff357aa43c3bede"
Oct 29 23:21:48.782970 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.782877    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f9dff7048aec5eb8ca721e37f148e3fb07db50bab779d76ff357aa43c3bede"} err="failed to get container status \"b1f9dff7048aec5eb8ca721e37f148e3fb07db50bab779d76ff357aa43c3bede\": rpc error: code = NotFound desc = could not find container \"b1f9dff7048aec5eb8ca721e37f148e3fb07db50bab779d76ff357aa43c3bede\": container with ID starting with b1f9dff7048aec5eb8ca721e37f148e3fb07db50bab779d76ff357aa43c3bede not found: ID does not exist"
Oct 29 23:21:48.786980 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.786946    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vs95c"]
Oct 29 23:21:48.841821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.833017    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"]
Oct 29 23:21:48.841821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:48.839320    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-x2wlh"]
Oct 29 23:21:49.940163 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:49.940123    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:50.778690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:50.778657    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eef2e22-9704-4291-b7ae-91c29acd11de" path="/var/lib/kubelet/pods/2eef2e22-9704-4291-b7ae-91c29acd11de/volumes"
Oct 29 23:21:50.779200 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:50.779185    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb" path="/var/lib/kubelet/pods/7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb/volumes"
Oct 29 23:21:51.900707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:51.900676    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:53.816059 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:53.816023    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:55.514861 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:55.514825    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:57.198122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.198091    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"]
Oct 29 23:21:57.198991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.198968    5428 cpu_manager.go:414] "containerMap: removing container" podUID="76bcb1e7-3c9e-4222-abc4-96c5c461707c" containerName="istio-proxy"
Oct 29 23:21:57.198991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.198988    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bcb1e7-3c9e-4222-abc4-96c5c461707c" containerName="istio-proxy"
Oct 29 23:21:57.199164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.198998    5428 cpu_manager.go:414] "containerMap: removing container" podUID="2eef2e22-9704-4291-b7ae-91c29acd11de" containerName="istio-proxy"
Oct 29 23:21:57.199164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.199004    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eef2e22-9704-4291-b7ae-91c29acd11de" containerName="istio-proxy"
Oct 29 23:21:57.199164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.199029    5428 cpu_manager.go:414] "containerMap: removing container" podUID="8e3f78e3-9804-4b4f-8e9d-6e37b0351643" containerName="istio-proxy"
Oct 29 23:21:57.199164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.199034    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3f78e3-9804-4b4f-8e9d-6e37b0351643" containerName="istio-proxy"
Oct 29 23:21:57.199164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.199045    5428 cpu_manager.go:414] "containerMap: removing container" podUID="dadecf77-4bd8-4ca8-bd71-3b2f36b62953" containerName="maas-api"
Oct 29 23:21:57.199164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.199050    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dadecf77-4bd8-4ca8-bd71-3b2f36b62953" containerName="maas-api"
Oct 29 23:21:57.199164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.199086    5428 cpu_manager.go:414] "containerMap: removing container" podUID="0de8513d-c8dc-4856-b5d1-8174e0676e1b" containerName="istio-proxy"
Oct 29 23:21:57.199164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.199091    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de8513d-c8dc-4856-b5d1-8174e0676e1b" containerName="istio-proxy"
Oct 29 23:21:57.199164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.199100    5428 cpu_manager.go:414] "containerMap: removing container" podUID="7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb" containerName="discovery"
Oct 29 23:21:57.199164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.199105    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb" containerName="discovery"
Oct 29 23:21:57.199727 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.199209    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de8513d-c8dc-4856-b5d1-8174e0676e1b" containerName="istio-proxy"
Oct 29 23:21:57.199727 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.199217    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3c6c27-9aa7-4242-b230-23d1ce4a2dcb" containerName="discovery"
Oct 29 23:21:57.199727 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.199225    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bcb1e7-3c9e-4222-abc4-96c5c461707c" containerName="istio-proxy"
Oct 29 23:21:57.199727 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.199230    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eef2e22-9704-4291-b7ae-91c29acd11de" containerName="istio-proxy"
Oct 29 23:21:57.199727 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.199240    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3f78e3-9804-4b4f-8e9d-6e37b0351643" containerName="istio-proxy"
Oct 29 23:21:57.199727 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.199247    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="dadecf77-4bd8-4ca8-bd71-3b2f36b62953" containerName="maas-api"
Oct 29 23:21:57.204501 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.204482    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"
Oct 29 23:21:57.219182 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.219161    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-206095-serving-cert\""
Oct 29 23:21:57.236269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.236239    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"]
Oct 29 23:21:57.297558 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.297528    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:57.324524 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.324492    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-maas-api-tls\") pod \"maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6\" (UID: \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"
Oct 29 23:21:57.324677 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.324549    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-metrics-certs\") pod \"maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6\" (UID: \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"
Oct 29 23:21:57.324765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.324751    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-555kf\" (UniqueName: \"kubernetes.io/projected/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-kube-api-access-555kf\") pod \"maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6\" (UID: \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"
Oct 29 23:21:57.324903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.324877    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6\" (UID: \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"
Oct 29 23:21:57.415373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.415339    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:57.426252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.426203    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-555kf\" (UniqueName: \"kubernetes.io/projected/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-kube-api-access-555kf\") pod \"maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6\" (UID: \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"
Oct 29 23:21:57.426417 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.426292    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6\" (UID: \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"
Oct 29 23:21:57.426417 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.426346    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-maas-api-tls\") pod \"maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6\" (UID: \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"
Oct 29 23:21:57.426417 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.426387    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-metrics-certs\") pod \"maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6\" (UID: \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"
Oct 29 23:21:57.427019 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.426988    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6\" (UID: \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"
Oct 29 23:21:57.428903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.428875    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-metrics-certs\") pod \"maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6\" (UID: \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"
Oct 29 23:21:57.429032 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.429009    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-maas-api-tls\") pod \"maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6\" (UID: \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"
Oct 29 23:21:57.457767 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.457645    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-555kf\" (UniqueName: \"kubernetes.io/projected/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-kube-api-access-555kf\") pod \"maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6\" (UID: \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"
Oct 29 23:21:57.519569 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.519531    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"
Oct 29 23:21:57.534907 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.534871    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:21:57.833089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.833059    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"]
Oct 29 23:21:57.834024 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:21:57.834003    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59b3e5a_dbbb_47e5_b2b1_67b9ef669dec.slice/crio-32403120b0295585caff37571b08df70867c4ac771854e5437c614c5baf7f1cb WatchSource:0}: Error finding container 32403120b0295585caff37571b08df70867c4ac771854e5437c614c5baf7f1cb: Status 404 returned error can't find the container with id 32403120b0295585caff37571b08df70867c4ac771854e5437c614c5baf7f1cb
Oct 29 23:21:57.835555 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:57.835538    5428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:21:58.807373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:58.807339    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6" event={"ID":"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec","Type":"ContainerStarted","Data":"fb1578059adf619a7ea04c1df22c33ec6d19e4af6985e236c44c652aefda0bab"}
Oct 29 23:21:58.807903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:58.807392    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6" event={"ID":"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec","Type":"ContainerStarted","Data":"32403120b0295585caff37571b08df70867c4ac771854e5437c614c5baf7f1cb"}
Oct 29 23:21:58.807903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:58.807566    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"
Oct 29 23:21:58.856100 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:58.856044    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6" podStartSLOduration=1.478858617 podStartE2EDuration="1.856030774s" totalImagesPullingTime="377.172157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:58.851921206 +0000 UTC m=+2322.194870021" watchObservedRunningTime="2026-10-29 23:21:58.856030774 +0000 UTC m=+2322.198979519"
Oct 29 23:21:59.458864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:21:59.458790    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:01.290193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:01.290163    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:01.713238 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:01.713201    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:02.179905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:02.179870    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:02.726896 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:02.726862    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:03.402093 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:03.402057    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:04.644895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:04.644855    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:04.820524 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:04.820497    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"
Oct 29 23:22:05.283389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:05.283342    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:06.219315 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:06.219277    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:06.688467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:06.688430    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:07.203138 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:07.203104    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:07.407005 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:07.406969    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:08.122352 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:08.122322    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:09.375893 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:09.375855    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:10.928875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:10.928844    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:11.065895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:11.065858    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:11.353210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:11.353129    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:11.534614 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:11.534575    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:11.940187 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:11.940152    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:12.537122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:12.537079    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:13.022986 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.022948    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:13.126333 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.126304    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269"]
Oct 29 23:22:13.130947 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.130922    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269"
Oct 29 23:22:13.149094 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.149069    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269"]
Oct 29 23:22:13.153923 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.153895    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:13.156286 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.156262    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-6f162a-dockercfg-lbppl\""
Oct 29 23:22:13.156442 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.156323    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-6f162a\""
Oct 29 23:22:13.194474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.194445    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf"]
Oct 29 23:22:13.200371 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.200346    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf"
Oct 29 23:22:13.220697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.220671    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf"]
Oct 29 23:22:13.305876 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.288147    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7e9b55-f400-4b97-ad21-b33e3354c297-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269\" (UID: \"6a7e9b55-f400-4b97-ad21-b33e3354c297\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269"
Oct 29 23:22:13.342759 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.342725    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnjg2\" (UniqueName: \"kubernetes.io/projected/6a7e9b55-f400-4b97-ad21-b33e3354c297-kube-api-access-wnjg2\") pod \"payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269\" (UID: \"6a7e9b55-f400-4b97-ad21-b33e3354c297\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269"
Oct 29 23:22:13.444242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.444199    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7e9b55-f400-4b97-ad21-b33e3354c297-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269\" (UID: \"6a7e9b55-f400-4b97-ad21-b33e3354c297\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269"
Oct 29 23:22:13.444449 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.444264    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnjg2\" (UniqueName: \"kubernetes.io/projected/6a7e9b55-f400-4b97-ad21-b33e3354c297-kube-api-access-wnjg2\") pod \"payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269\" (UID: \"6a7e9b55-f400-4b97-ad21-b33e3354c297\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269"
Oct 29 23:22:13.444449 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.444362    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78dd7d7-43b5-4e61-96b9-b040ab1185ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf\" (UID: \"e78dd7d7-43b5-4e61-96b9-b040ab1185ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf"
Oct 29 23:22:13.444449 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.444400    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77gtl\" (UniqueName: \"kubernetes.io/projected/e78dd7d7-43b5-4e61-96b9-b040ab1185ed-kube-api-access-77gtl\") pod \"payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf\" (UID: \"e78dd7d7-43b5-4e61-96b9-b040ab1185ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf"
Oct 29 23:22:13.444943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.444923    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7e9b55-f400-4b97-ad21-b33e3354c297-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269\" (UID: \"6a7e9b55-f400-4b97-ad21-b33e3354c297\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269"
Oct 29 23:22:13.467615 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.467584    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnjg2\" (UniqueName: \"kubernetes.io/projected/6a7e9b55-f400-4b97-ad21-b33e3354c297-kube-api-access-wnjg2\") pod \"payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269\" (UID: \"6a7e9b55-f400-4b97-ad21-b33e3354c297\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269"
Oct 29 23:22:13.546482 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.546444    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78dd7d7-43b5-4e61-96b9-b040ab1185ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf\" (UID: \"e78dd7d7-43b5-4e61-96b9-b040ab1185ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf"
Oct 29 23:22:13.546654 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.546499    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77gtl\" (UniqueName: \"kubernetes.io/projected/e78dd7d7-43b5-4e61-96b9-b040ab1185ed-kube-api-access-77gtl\") pod \"payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf\" (UID: \"e78dd7d7-43b5-4e61-96b9-b040ab1185ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf"
Oct 29 23:22:13.547366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.547340    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78dd7d7-43b5-4e61-96b9-b040ab1185ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf\" (UID: \"e78dd7d7-43b5-4e61-96b9-b040ab1185ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf"
Oct 29 23:22:13.570232 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.570159    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77gtl\" (UniqueName: \"kubernetes.io/projected/e78dd7d7-43b5-4e61-96b9-b040ab1185ed-kube-api-access-77gtl\") pod \"payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf\" (UID: \"e78dd7d7-43b5-4e61-96b9-b040ab1185ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf"
Oct 29 23:22:13.747618 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.747584    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269"
Oct 29 23:22:13.816920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:13.816848    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf"
Oct 29 23:22:14.065491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:14.065462    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269"]
Oct 29 23:22:14.065966 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:22:14.065941    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7e9b55_f400_4b97_ad21_b33e3354c297.slice/crio-8ee5ec03dd0f527679f970bf12b6c61d75d08cc82998b09a5b0418af3b88282f WatchSource:0}: Error finding container 8ee5ec03dd0f527679f970bf12b6c61d75d08cc82998b09a5b0418af3b88282f: Status 404 returned error can't find the container with id 8ee5ec03dd0f527679f970bf12b6c61d75d08cc82998b09a5b0418af3b88282f
Oct 29 23:22:14.164349 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:14.164324    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf"]
Oct 29 23:22:14.165368 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:22:14.165340    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78dd7d7_43b5_4e61_96b9_b040ab1185ed.slice/crio-d6c4c8f93bec29c08d17f5d12ac6155c97995df0777ad3b8a048415efae9ed8e WatchSource:0}: Error finding container d6c4c8f93bec29c08d17f5d12ac6155c97995df0777ad3b8a048415efae9ed8e: Status 404 returned error can't find the container with id d6c4c8f93bec29c08d17f5d12ac6155c97995df0777ad3b8a048415efae9ed8e
Oct 29 23:22:14.175555 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:14.175521    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:14.954386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:14.954348    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf" event={"ID":"e78dd7d7-43b5-4e61-96b9-b040ab1185ed","Type":"ContainerStarted","Data":"ad9e5c82f6e1f5851a4ce6f1dec56f95d3620f178b541708649ecee922d5f4a8"}
Oct 29 23:22:14.954584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:14.954405    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf" event={"ID":"e78dd7d7-43b5-4e61-96b9-b040ab1185ed","Type":"ContainerStarted","Data":"d6c4c8f93bec29c08d17f5d12ac6155c97995df0777ad3b8a048415efae9ed8e"}
Oct 29 23:22:14.954584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:14.954551    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf"
Oct 29 23:22:14.956252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:14.956227    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269" event={"ID":"6a7e9b55-f400-4b97-ad21-b33e3354c297","Type":"ContainerStarted","Data":"9244cca83282379d084210fe84a7d68a09dfef081d8f427ab8b6510e4c66abf8"}
Oct 29 23:22:14.956407 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:14.956260    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269" event={"ID":"6a7e9b55-f400-4b97-ad21-b33e3354c297","Type":"ContainerStarted","Data":"8ee5ec03dd0f527679f970bf12b6c61d75d08cc82998b09a5b0418af3b88282f"}
Oct 29 23:22:14.956407 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:14.956306    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269"
Oct 29 23:22:14.994938 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:14.994885    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf" podStartSLOduration=1.994871803 podStartE2EDuration="1.994871803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:14.99433977 +0000 UTC m=+2338.337288617" watchObservedRunningTime="2026-10-29 23:22:14.994871803 +0000 UTC m=+2338.337820568"
Oct 29 23:22:15.035987 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:15.035929    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269" podStartSLOduration=2.035915091 podStartE2EDuration="2.035915091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:15.033289551 +0000 UTC m=+2338.376238333" watchObservedRunningTime="2026-10-29 23:22:15.035915091 +0000 UTC m=+2338.378863835"
Oct 29 23:22:16.107289 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:16.107255    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:16.844010 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:16.843978    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:16.977226 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:16.977195    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:17.287997 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:17.287964    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:17.424642 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:22:17.424548    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:22:18.317035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:18.317005    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:18.791312 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:18.791265    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:18.953376 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:18.953346    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:19.063442 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.063366    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf"]
Oct 29 23:22:19.069968 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.069936    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf"
Oct 29 23:22:19.088790 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.088758    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-c1e164\""
Oct 29 23:22:19.091593 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.091566    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:19.093374 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.093351    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf"]
Oct 29 23:22:19.095722 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.095698    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-c1e164-dockercfg-4djbw\""
Oct 29 23:22:19.112395 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.112364    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8"]
Oct 29 23:22:19.118794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.118766    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8"
Oct 29 23:22:19.146100 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.146071    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8"]
Oct 29 23:22:19.231955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.231918    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlpwm\" (UniqueName: \"kubernetes.io/projected/07b3da27-929f-471c-a627-1ac5e4af844f-kube-api-access-rlpwm\") pod \"payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8\" (UID: \"07b3da27-929f-471c-a627-1ac5e4af844f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8"
Oct 29 23:22:19.232156 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.232074    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9frph\" (UniqueName: \"kubernetes.io/projected/fd5901f6-73be-4fd8-b7c1-d9237b6e934a-kube-api-access-9frph\") pod \"payload-processing-e2e-autores-c1e164-6f7996656-pw8tf\" (UID: \"fd5901f6-73be-4fd8-b7c1-d9237b6e934a\") " pod="openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf"
Oct 29 23:22:19.232156 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.232135    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b3da27-929f-471c-a627-1ac5e4af844f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8\" (UID: \"07b3da27-929f-471c-a627-1ac5e4af844f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8"
Oct 29 23:22:19.232292 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.232259    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5901f6-73be-4fd8-b7c1-d9237b6e934a-plugins-config-volume\") pod \"payload-processing-e2e-autores-c1e164-6f7996656-pw8tf\" (UID: \"fd5901f6-73be-4fd8-b7c1-d9237b6e934a\") " pod="openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf"
Oct 29 23:22:19.263694 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.263668    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:19.333766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.333674    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlpwm\" (UniqueName: \"kubernetes.io/projected/07b3da27-929f-471c-a627-1ac5e4af844f-kube-api-access-rlpwm\") pod \"payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8\" (UID: \"07b3da27-929f-471c-a627-1ac5e4af844f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8"
Oct 29 23:22:19.334488 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.334458    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9frph\" (UniqueName: \"kubernetes.io/projected/fd5901f6-73be-4fd8-b7c1-d9237b6e934a-kube-api-access-9frph\") pod \"payload-processing-e2e-autores-c1e164-6f7996656-pw8tf\" (UID: \"fd5901f6-73be-4fd8-b7c1-d9237b6e934a\") " pod="openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf"
Oct 29 23:22:19.334624 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.334550    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b3da27-929f-471c-a627-1ac5e4af844f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8\" (UID: \"07b3da27-929f-471c-a627-1ac5e4af844f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8"
Oct 29 23:22:19.334704 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.334638    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5901f6-73be-4fd8-b7c1-d9237b6e934a-plugins-config-volume\") pod \"payload-processing-e2e-autores-c1e164-6f7996656-pw8tf\" (UID: \"fd5901f6-73be-4fd8-b7c1-d9237b6e934a\") " pod="openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf"
Oct 29 23:22:19.335565 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.335537    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b3da27-929f-471c-a627-1ac5e4af844f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8\" (UID: \"07b3da27-929f-471c-a627-1ac5e4af844f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8"
Oct 29 23:22:19.335671 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.335626    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5901f6-73be-4fd8-b7c1-d9237b6e934a-plugins-config-volume\") pod \"payload-processing-e2e-autores-c1e164-6f7996656-pw8tf\" (UID: \"fd5901f6-73be-4fd8-b7c1-d9237b6e934a\") " pod="openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf"
Oct 29 23:22:19.406955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.406916    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9frph\" (UniqueName: \"kubernetes.io/projected/fd5901f6-73be-4fd8-b7c1-d9237b6e934a-kube-api-access-9frph\") pod \"payload-processing-e2e-autores-c1e164-6f7996656-pw8tf\" (UID: \"fd5901f6-73be-4fd8-b7c1-d9237b6e934a\") " pod="openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf"
Oct 29 23:22:19.411285 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.411254    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:19.416195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.414410    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlpwm\" (UniqueName: \"kubernetes.io/projected/07b3da27-929f-471c-a627-1ac5e4af844f-kube-api-access-rlpwm\") pod \"payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8\" (UID: \"07b3da27-929f-471c-a627-1ac5e4af844f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8"
Oct 29 23:22:19.447108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.447072    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8"
Oct 29 23:22:19.694032 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.694000    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf"
Oct 29 23:22:19.772882 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.772849    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8"]
Oct 29 23:22:19.997917 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.997825    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8" event={"ID":"07b3da27-929f-471c-a627-1ac5e4af844f","Type":"ContainerStarted","Data":"e7d96d69884541d6297bb0ec6dda9bd64b81fac73b9fa822e6da7377ce8cc9e7"}
Oct 29 23:22:19.997917 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.997885    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8" event={"ID":"07b3da27-929f-471c-a627-1ac5e4af844f","Type":"ContainerStarted","Data":"c3e2b93a332f019451fecb19db9ee1089b2d2c5288a25cb51d844c69b4e9f9ee"}
Oct 29 23:22:19.998107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:19.997941    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8"
Oct 29 23:22:20.008868 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:20.008844    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf"]
Oct 29 23:22:20.009707 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:22:20.009686    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5901f6_73be_4fd8_b7c1_d9237b6e934a.slice/crio-01d81a2be912941f87523254b84d60ed56f2782d31f97d7bbc5edd49ee7a8554 WatchSource:0}: Error finding container 01d81a2be912941f87523254b84d60ed56f2782d31f97d7bbc5edd49ee7a8554: Status 404 returned error can't find the container with id 01d81a2be912941f87523254b84d60ed56f2782d31f97d7bbc5edd49ee7a8554
Oct 29 23:22:20.043497 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:20.043454    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8" podStartSLOduration=1.043439685 podStartE2EDuration="1.043439685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:20.042067282 +0000 UTC m=+2343.385016027" watchObservedRunningTime="2026-10-29 23:22:20.043439685 +0000 UTC m=+2343.386388432"
Oct 29 23:22:20.432571 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:20.432539    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:21.007470 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:21.007431    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf" event={"ID":"fd5901f6-73be-4fd8-b7c1-d9237b6e934a","Type":"ContainerStarted","Data":"4c95d3c055065e02d005397f3c0a6aa8b7a7cfadaa26b2790e71de4da2ee5fba"}
Oct 29 23:22:21.007679 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:21.007503    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf"
Oct 29 23:22:21.007679 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:21.007527    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf" event={"ID":"fd5901f6-73be-4fd8-b7c1-d9237b6e934a","Type":"ContainerStarted","Data":"01d81a2be912941f87523254b84d60ed56f2782d31f97d7bbc5edd49ee7a8554"}
Oct 29 23:22:21.050680 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:21.050631    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf" podStartSLOduration=2.050618241 podStartE2EDuration="2.050618241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:21.047984657 +0000 UTC m=+2344.390933446" watchObservedRunningTime="2026-10-29 23:22:21.050618241 +0000 UTC m=+2344.393567008"
Oct 29 23:22:22.609751 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:22.609721    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:22.997034 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:22.997002    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:23.150282 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:23.150239    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:23.270922 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:23.270840    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:24.597630 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:24.597597    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:24.714071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:24.714040    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:25.079817 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.079767    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:25.246849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.237001    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:25.543339 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.543296    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:25.642045 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.642009    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br"]
Oct 29 23:22:25.648338 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.648317    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br"
Oct 29 23:22:25.665056 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.665026    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br"]
Oct 29 23:22:25.669717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.669692    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-e10975\""
Oct 29 23:22:25.669898 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.669879    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:25.681115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.681084    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-e10975-dockercfg-wwk4w\""
Oct 29 23:22:25.703649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.703618    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m"]
Oct 29 23:22:25.710067 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.710041    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m"
Oct 29 23:22:25.723718 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.723687    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m"]
Oct 29 23:22:25.745172 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.745121    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7850180b-94dc-4e7b-9323-b2ba77d1884d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br\" (UID: \"7850180b-94dc-4e7b-9323-b2ba77d1884d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br"
Oct 29 23:22:25.746597 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.745782    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qftgd\" (UniqueName: \"kubernetes.io/projected/7850180b-94dc-4e7b-9323-b2ba77d1884d-kube-api-access-qftgd\") pod \"payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br\" (UID: \"7850180b-94dc-4e7b-9323-b2ba77d1884d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br"
Oct 29 23:22:25.853530 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.853400    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7850180b-94dc-4e7b-9323-b2ba77d1884d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br\" (UID: \"7850180b-94dc-4e7b-9323-b2ba77d1884d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br"
Oct 29 23:22:25.858357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.858318    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5tvx\" (UniqueName: \"kubernetes.io/projected/984a1e35-8988-450a-80ad-76b3bfe60f38-kube-api-access-p5tvx\") pod \"payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m\" (UID: \"984a1e35-8988-450a-80ad-76b3bfe60f38\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m"
Oct 29 23:22:25.858744 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.858718    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qftgd\" (UniqueName: \"kubernetes.io/projected/7850180b-94dc-4e7b-9323-b2ba77d1884d-kube-api-access-qftgd\") pod \"payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br\" (UID: \"7850180b-94dc-4e7b-9323-b2ba77d1884d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br"
Oct 29 23:22:25.859300 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.859269    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984a1e35-8988-450a-80ad-76b3bfe60f38-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m\" (UID: \"984a1e35-8988-450a-80ad-76b3bfe60f38\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m"
Oct 29 23:22:25.859615 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.857721    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7850180b-94dc-4e7b-9323-b2ba77d1884d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br\" (UID: \"7850180b-94dc-4e7b-9323-b2ba77d1884d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br"
Oct 29 23:22:25.937652 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.937485    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qftgd\" (UniqueName: \"kubernetes.io/projected/7850180b-94dc-4e7b-9323-b2ba77d1884d-kube-api-access-qftgd\") pod \"payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br\" (UID: \"7850180b-94dc-4e7b-9323-b2ba77d1884d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br"
Oct 29 23:22:25.960629 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.960588    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5tvx\" (UniqueName: \"kubernetes.io/projected/984a1e35-8988-450a-80ad-76b3bfe60f38-kube-api-access-p5tvx\") pod \"payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m\" (UID: \"984a1e35-8988-450a-80ad-76b3bfe60f38\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m"
Oct 29 23:22:25.960866 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.960761    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984a1e35-8988-450a-80ad-76b3bfe60f38-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m\" (UID: \"984a1e35-8988-450a-80ad-76b3bfe60f38\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m"
Oct 29 23:22:25.961579 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.961548    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984a1e35-8988-450a-80ad-76b3bfe60f38-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m\" (UID: \"984a1e35-8988-450a-80ad-76b3bfe60f38\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m"
Oct 29 23:22:25.968079 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.968014    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf"
Oct 29 23:22:25.968489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.968416    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269"
Oct 29 23:22:25.972901 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.970408    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br"
Oct 29 23:22:25.989549 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:25.986657    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5tvx\" (UniqueName: \"kubernetes.io/projected/984a1e35-8988-450a-80ad-76b3bfe60f38-kube-api-access-p5tvx\") pod \"payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m\" (UID: \"984a1e35-8988-450a-80ad-76b3bfe60f38\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m"
Oct 29 23:22:26.045823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:26.045060    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m"
Oct 29 23:22:26.342004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:26.341971    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br"]
Oct 29 23:22:26.349925 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:22:26.349891    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7850180b_94dc_4e7b_9323_b2ba77d1884d.slice/crio-af062bdb5e2cba869896edfed55bc2a429a041800e7b79bcce50a537b7fccce3 WatchSource:0}: Error finding container af062bdb5e2cba869896edfed55bc2a429a041800e7b79bcce50a537b7fccce3: Status 404 returned error can't find the container with id af062bdb5e2cba869896edfed55bc2a429a041800e7b79bcce50a537b7fccce3
Oct 29 23:22:26.471337 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:26.471305    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m"]
Oct 29 23:22:26.471917 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:22:26.471887    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984a1e35_8988_450a_80ad_76b3bfe60f38.slice/crio-de29f328d98e5083daa19301b43bcf515705a5c557c3d542fc795f82073c2a38 WatchSource:0}: Error finding container de29f328d98e5083daa19301b43bcf515705a5c557c3d542fc795f82073c2a38: Status 404 returned error can't find the container with id de29f328d98e5083daa19301b43bcf515705a5c557c3d542fc795f82073c2a38
Oct 29 23:22:27.091382 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.091332    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m" event={"ID":"984a1e35-8988-450a-80ad-76b3bfe60f38","Type":"ContainerStarted","Data":"29f1b54421724b2a054c1c763dfbc1aa6546cec1ae05f8c3a1f3dfee3e0137f2"}
Oct 29 23:22:27.092050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.092022    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m" event={"ID":"984a1e35-8988-450a-80ad-76b3bfe60f38","Type":"ContainerStarted","Data":"de29f328d98e5083daa19301b43bcf515705a5c557c3d542fc795f82073c2a38"}
Oct 29 23:22:27.092511 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.092489    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m"
Oct 29 23:22:27.095382 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.095351    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br" event={"ID":"7850180b-94dc-4e7b-9323-b2ba77d1884d","Type":"ContainerStarted","Data":"45f9a74cc7fc6d2d7ee2ef8f9777dc104219d995d5878cfd28ee841542cacc34"}
Oct 29 23:22:27.095560 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.095543    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br" event={"ID":"7850180b-94dc-4e7b-9323-b2ba77d1884d","Type":"ContainerStarted","Data":"af062bdb5e2cba869896edfed55bc2a429a041800e7b79bcce50a537b7fccce3"}
Oct 29 23:22:27.134659 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.134611    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m" podStartSLOduration=2.13459848 podStartE2EDuration="2.13459848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:27.134083368 +0000 UTC m=+2350.477032114" watchObservedRunningTime="2026-10-29 23:22:27.13459848 +0000 UTC m=+2350.477547225"
Oct 29 23:22:27.177958 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.177902    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br" podStartSLOduration=2.177885624 podStartE2EDuration="2.177885624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:27.173888771 +0000 UTC m=+2350.516837543" watchObservedRunningTime="2026-10-29 23:22:27.177885624 +0000 UTC m=+2350.520834370"
Oct 29 23:22:27.323141 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.323107    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"]
Oct 29 23:22:27.328321 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.328298    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"
Oct 29 23:22:27.347445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.347343    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-a84df8-serving-cert\""
Oct 29 23:22:27.347865 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.347835    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"]
Oct 29 23:22:27.399141 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.399098    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d16a34-4567-4586-a992-297386786fad-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp\" (UID: \"64d16a34-4567-4586-a992-297386786fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"
Oct 29 23:22:27.399329 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.399181    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d16a34-4567-4586-a992-297386786fad-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp\" (UID: \"64d16a34-4567-4586-a992-297386786fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"
Oct 29 23:22:27.399481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.399456    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d16a34-4567-4586-a992-297386786fad-metrics-certs\") pod \"maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp\" (UID: \"64d16a34-4567-4586-a992-297386786fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"
Oct 29 23:22:27.399649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.399629    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9m8v\" (UniqueName: \"kubernetes.io/projected/64d16a34-4567-4586-a992-297386786fad-kube-api-access-x9m8v\") pod \"maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp\" (UID: \"64d16a34-4567-4586-a992-297386786fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"
Oct 29 23:22:27.427239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.427205    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:27.501185 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.501151    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d16a34-4567-4586-a992-297386786fad-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp\" (UID: \"64d16a34-4567-4586-a992-297386786fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"
Oct 29 23:22:27.501377 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.501205    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d16a34-4567-4586-a992-297386786fad-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp\" (UID: \"64d16a34-4567-4586-a992-297386786fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"
Oct 29 23:22:27.501439 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.501412    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d16a34-4567-4586-a992-297386786fad-metrics-certs\") pod \"maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp\" (UID: \"64d16a34-4567-4586-a992-297386786fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"
Oct 29 23:22:27.501568 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.501548    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9m8v\" (UniqueName: \"kubernetes.io/projected/64d16a34-4567-4586-a992-297386786fad-kube-api-access-x9m8v\") pod \"maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp\" (UID: \"64d16a34-4567-4586-a992-297386786fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"
Oct 29 23:22:27.502157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.502127    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d16a34-4567-4586-a992-297386786fad-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp\" (UID: \"64d16a34-4567-4586-a992-297386786fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"
Oct 29 23:22:27.503892 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.503868    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d16a34-4567-4586-a992-297386786fad-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp\" (UID: \"64d16a34-4567-4586-a992-297386786fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"
Oct 29 23:22:27.504001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.503920    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d16a34-4567-4586-a992-297386786fad-metrics-certs\") pod \"maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp\" (UID: \"64d16a34-4567-4586-a992-297386786fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"
Oct 29 23:22:27.528479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.528446    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9m8v\" (UniqueName: \"kubernetes.io/projected/64d16a34-4567-4586-a992-297386786fad-kube-api-access-x9m8v\") pod \"maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp\" (UID: \"64d16a34-4567-4586-a992-297386786fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"
Oct 29 23:22:27.547127 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.547092    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:27.651045 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.651012    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"
Oct 29 23:22:27.993938 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:27.993851    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"]
Oct 29 23:22:28.110087 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:28.110048    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp" event={"ID":"64d16a34-4567-4586-a992-297386786fad","Type":"ContainerStarted","Data":"24bb118b9749df735a74d14c148915da2827eefbd784d75f5c8ea1ed085545bc"}
Oct 29 23:22:28.110628 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:28.110522    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br"
Oct 29 23:22:29.131075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:29.131040    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp" event={"ID":"64d16a34-4567-4586-a992-297386786fad","Type":"ContainerStarted","Data":"177fcfe0668586cd1ba36e2b2f15517ea5eb552b07d6077d65db0f6d7680039e"}
Oct 29 23:22:29.131665 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:29.131645    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"
Oct 29 23:22:29.220525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:29.219739    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp" podStartSLOduration=1.828203898 podStartE2EDuration="2.219718025s" totalImagesPullingTime="391.514127ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:29.180105629 +0000 UTC m=+2352.523054375" watchObservedRunningTime="2026-10-29 23:22:29.219718025 +0000 UTC m=+2352.562666771"
Oct 29 23:22:29.499880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:29.499200    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:30.185824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:30.185576    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:30.440771 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:30.440668    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:30.593964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:30.593922    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:31.011507 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:31.011473    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8"
Oct 29 23:22:31.910581 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:31.910546    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:32.018601 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.018568    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf"
Oct 29 23:22:32.020226 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.020195    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq"]
Oct 29 23:22:32.025534 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.025513    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq"
Oct 29 23:22:32.043230 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.043197    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:32.047710 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.047683    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c5ef2f\""
Oct 29 23:22:32.053226 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.053192    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq"]
Oct 29 23:22:32.090271 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.090027    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5186bf82-c9e1-49e3-9dd0-f362ddce36ce-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq\" (UID: \"5186bf82-c9e1-49e3-9dd0-f362ddce36ce\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq"
Oct 29 23:22:32.090271 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.090176    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdw5f\" (UniqueName: \"kubernetes.io/projected/5186bf82-c9e1-49e3-9dd0-f362ddce36ce-kube-api-access-tdw5f\") pod \"payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq\" (UID: \"5186bf82-c9e1-49e3-9dd0-f362ddce36ce\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq"
Oct 29 23:22:32.098590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.098559    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv"]
Oct 29 23:22:32.105919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.105892    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv"
Oct 29 23:22:32.121418 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.121374    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c5ef2f-dockercfg-xs5q8\""
Oct 29 23:22:32.123729 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.123699    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv"]
Oct 29 23:22:32.192000 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.191909    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5186bf82-c9e1-49e3-9dd0-f362ddce36ce-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq\" (UID: \"5186bf82-c9e1-49e3-9dd0-f362ddce36ce\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq"
Oct 29 23:22:32.192000 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.191963    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdw5f\" (UniqueName: \"kubernetes.io/projected/5186bf82-c9e1-49e3-9dd0-f362ddce36ce-kube-api-access-tdw5f\") pod \"payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq\" (UID: \"5186bf82-c9e1-49e3-9dd0-f362ddce36ce\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq"
Oct 29 23:22:32.192272 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.192123    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d42d182-5615-4c04-a4af-5d7aee1c4dba-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv\" (UID: \"8d42d182-5615-4c04-a4af-5d7aee1c4dba\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv"
Oct 29 23:22:32.192272 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.192211    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv572\" (UniqueName: \"kubernetes.io/projected/8d42d182-5615-4c04-a4af-5d7aee1c4dba-kube-api-access-jv572\") pod \"payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv\" (UID: \"8d42d182-5615-4c04-a4af-5d7aee1c4dba\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv"
Oct 29 23:22:32.192954 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.192929    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5186bf82-c9e1-49e3-9dd0-f362ddce36ce-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq\" (UID: \"5186bf82-c9e1-49e3-9dd0-f362ddce36ce\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq"
Oct 29 23:22:32.289828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.286590    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdw5f\" (UniqueName: \"kubernetes.io/projected/5186bf82-c9e1-49e3-9dd0-f362ddce36ce-kube-api-access-tdw5f\") pod \"payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq\" (UID: \"5186bf82-c9e1-49e3-9dd0-f362ddce36ce\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq"
Oct 29 23:22:32.303819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.299934    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv572\" (UniqueName: \"kubernetes.io/projected/8d42d182-5615-4c04-a4af-5d7aee1c4dba-kube-api-access-jv572\") pod \"payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv\" (UID: \"8d42d182-5615-4c04-a4af-5d7aee1c4dba\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv"
Oct 29 23:22:32.303819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.300183    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d42d182-5615-4c04-a4af-5d7aee1c4dba-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv\" (UID: \"8d42d182-5615-4c04-a4af-5d7aee1c4dba\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv"
Oct 29 23:22:32.303819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.301157    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d42d182-5615-4c04-a4af-5d7aee1c4dba-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv\" (UID: \"8d42d182-5615-4c04-a4af-5d7aee1c4dba\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv"
Oct 29 23:22:32.330825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.330516    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv572\" (UniqueName: \"kubernetes.io/projected/8d42d182-5615-4c04-a4af-5d7aee1c4dba-kube-api-access-jv572\") pod \"payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv\" (UID: \"8d42d182-5615-4c04-a4af-5d7aee1c4dba\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv"
Oct 29 23:22:32.343934 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.343902    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq"
Oct 29 23:22:32.430488 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.430446    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv"
Oct 29 23:22:32.668119 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.668092    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq"]
Oct 29 23:22:32.669406 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:22:32.669369    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5186bf82_c9e1_49e3_9dd0_f362ddce36ce.slice/crio-14f55f69dd3864c0e63d2eaa5298ab0e9cf9fdf277e213bf2354f047d3c04c5e WatchSource:0}: Error finding container 14f55f69dd3864c0e63d2eaa5298ab0e9cf9fdf277e213bf2354f047d3c04c5e: Status 404 returned error can't find the container with id 14f55f69dd3864c0e63d2eaa5298ab0e9cf9fdf277e213bf2354f047d3c04c5e
Oct 29 23:22:32.776172 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.776127    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:32.788502 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:22:32.788468    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d42d182_5615_4c04_a4af_5d7aee1c4dba.slice/crio-c0ecf0eb7e7cdfa16d2da8be80cc7ed91e59480fb66016d90efffee17d0d30e9 WatchSource:0}: Error finding container c0ecf0eb7e7cdfa16d2da8be80cc7ed91e59480fb66016d90efffee17d0d30e9: Status 404 returned error can't find the container with id c0ecf0eb7e7cdfa16d2da8be80cc7ed91e59480fb66016d90efffee17d0d30e9
Oct 29 23:22:32.790110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:32.790079    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv"]
Oct 29 23:22:33.171498 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:33.171458    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq" event={"ID":"5186bf82-c9e1-49e3-9dd0-f362ddce36ce","Type":"ContainerStarted","Data":"86e21cfb98f5cf2370c3a3c36fdc7fe6f72183b6ed4e332e36f6d9712ed929cf"}
Oct 29 23:22:33.172027 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:33.171524    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq" event={"ID":"5186bf82-c9e1-49e3-9dd0-f362ddce36ce","Type":"ContainerStarted","Data":"14f55f69dd3864c0e63d2eaa5298ab0e9cf9fdf277e213bf2354f047d3c04c5e"}
Oct 29 23:22:33.172027 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:33.171557    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq"
Oct 29 23:22:33.173458 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:33.173426    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv" event={"ID":"8d42d182-5615-4c04-a4af-5d7aee1c4dba","Type":"ContainerStarted","Data":"490e845a407834d6ff17eaa047e1e2ac5a2d46f8bbddbaf0baa4db57ca290642"}
Oct 29 23:22:33.173619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:33.173476    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv" event={"ID":"8d42d182-5615-4c04-a4af-5d7aee1c4dba","Type":"ContainerStarted","Data":"c0ecf0eb7e7cdfa16d2da8be80cc7ed91e59480fb66016d90efffee17d0d30e9"}
Oct 29 23:22:33.173619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:33.173517    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv"
Oct 29 23:22:33.250370 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:33.250312    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq" podStartSLOduration=2.250297798 podStartE2EDuration="2.250297798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:33.246110996 +0000 UTC m=+2356.589059742" watchObservedRunningTime="2026-10-29 23:22:33.250297798 +0000 UTC m=+2356.593246587"
Oct 29 23:22:33.304228 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:33.304173    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv" podStartSLOduration=1.30415437 podStartE2EDuration="1.30415437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:33.301845508 +0000 UTC m=+2356.644794255" watchObservedRunningTime="2026-10-29 23:22:33.30415437 +0000 UTC m=+2356.647103118"
Oct 29 23:22:34.794970 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:34.794938    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:36.154921 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:36.154889    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"
Oct 29 23:22:36.578573 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:36.578490    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:38.114494 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:38.114465    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m"
Oct 29 23:22:38.405828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:38.405738    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:39.135161 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:39.135132    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br"
Oct 29 23:22:40.266558 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:40.266528    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:41.970384 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:41.970347    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:43.720371 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:43.720339    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:44.185377 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:44.185347    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq"
Oct 29 23:22:44.185576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:44.185415    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv"
Oct 29 23:22:45.570317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:45.570280    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:47.351190 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:47.351155    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:47.370879 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:22:47.370839    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:22:49.182246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:49.182213    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:50.831885 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.831844    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"]
Oct 29 23:22:50.837609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.837587    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.852707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.852683    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-2126fb-openshift-default-dockercfg-2swzx\""
Oct 29 23:22:50.869428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.869404    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"]
Oct 29 23:22:50.895886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.895851    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe45dacc-3544-4123-a779-db0455a8a702-istio-token\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.896041 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.895903    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe45dacc-3544-4123-a779-db0455a8a702-istiod-ca-cert\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.896116 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.896037    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7jxs\" (UniqueName: \"kubernetes.io/projected/fe45dacc-3544-4123-a779-db0455a8a702-kube-api-access-w7jxs\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.896116 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.896108    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-credential-socket\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.896245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.896221    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-workload-certs\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.896307 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.896259    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-istio-envoy\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.896307 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.896275    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-istio-data\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.896449 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.896432    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-workload-socket\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.896549 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.896527    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe45dacc-3544-4123-a779-db0455a8a702-istio-podinfo\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.997460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.997415    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe45dacc-3544-4123-a779-db0455a8a702-istio-token\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.997460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.997469    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe45dacc-3544-4123-a779-db0455a8a702-istiod-ca-cert\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.997678 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.997510    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7jxs\" (UniqueName: \"kubernetes.io/projected/fe45dacc-3544-4123-a779-db0455a8a702-kube-api-access-w7jxs\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.997678 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.997549    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-credential-socket\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.997678 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.997609    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-workload-certs\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.997678 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.997635    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-istio-envoy\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.997678 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.997659    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-istio-data\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.997982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.997742    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-workload-socket\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.997982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.997858    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe45dacc-3544-4123-a779-db0455a8a702-istio-podinfo\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.998100 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.998045    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-credential-socket\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.998161 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.998128    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-istio-data\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.998307 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.998285    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-workload-certs\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.998413 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.998290    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe45dacc-3544-4123-a779-db0455a8a702-istiod-ca-cert\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:50.998413 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:50.998341    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-workload-socket\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:51.000072 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:51.000046    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-istio-envoy\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:51.000264 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:51.000247    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe45dacc-3544-4123-a779-db0455a8a702-istio-podinfo\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:51.020730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:51.020697    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe45dacc-3544-4123-a779-db0455a8a702-istio-token\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:51.020875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:51.020765    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7jxs\" (UniqueName: \"kubernetes.io/projected/fe45dacc-3544-4123-a779-db0455a8a702-kube-api-access-w7jxs\") pod \"e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") " pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:51.157108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:51.157078    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:51.215252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:51.215226    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:51.490356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:51.490323    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"]
Oct 29 23:22:51.490853 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:22:51.490827    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe45dacc_3544_4123_a779_db0455a8a702.slice/crio-2dd9140b4d9743feab861e2924e768c24a7f221e59776c83b4df21dee53764ef WatchSource:0}: Error finding container 2dd9140b4d9743feab861e2924e768c24a7f221e59776c83b4df21dee53764ef: Status 404 returned error can't find the container with id 2dd9140b4d9743feab861e2924e768c24a7f221e59776c83b4df21dee53764ef
Oct 29 23:22:51.494171 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:51.494136    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:22:51.494293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:51.494226    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:22:51.494293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:51.494277    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:22:52.343736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:52.343700    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc" event={"ID":"fe45dacc-3544-4123-a779-db0455a8a702","Type":"ContainerStarted","Data":"d4c86113710102dde56aeceec7a58779c511bcca03b39cf5cdaff2a936a43506"}
Oct 29 23:22:52.344165 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:52.343756    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc" event={"ID":"fe45dacc-3544-4123-a779-db0455a8a702","Type":"ContainerStarted","Data":"2dd9140b4d9743feab861e2924e768c24a7f221e59776c83b4df21dee53764ef"}
Oct 29 23:22:52.404545 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:52.404486    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc" podStartSLOduration=2.404470225 podStartE2EDuration="2.404470225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:52.400666383 +0000 UTC m=+2375.743615129" watchObservedRunningTime="2026-10-29 23:22:52.404470225 +0000 UTC m=+2375.747419042"
Oct 29 23:22:53.084959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:53.084924    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:53.158232 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:53.158192    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:53.179993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:53.179951    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 29 23:22:53.180694 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:53.180659    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc" podUID="fe45dacc-3544-4123-a779-db0455a8a702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 29 23:22:54.157909 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:54.157882    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 29 23:22:54.158290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:54.157926    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc" podUID="fe45dacc-3544-4123-a779-db0455a8a702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 29 23:22:54.187847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:54.187588    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:54.810529 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:54.809979    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:55.158202 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:55.158154    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 29 23:22:55.158725 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:55.158225    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc" podUID="fe45dacc-3544-4123-a779-db0455a8a702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 29 23:22:55.385339 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:55.385309    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:55.512483 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:55.512356    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:55.784298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:55.784208    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:55.987459 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:55.987419    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:56.136769 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:56.136734    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:56.157810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:56.157762    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 29 23:22:56.157967 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:56.157845    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc" podUID="fe45dacc-3544-4123-a779-db0455a8a702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 29 23:22:56.254330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:56.254292    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:56.796775 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:56.796154    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:56.945260 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:56.945221    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:57.158551 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:57.158512    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 29 23:22:57.158748 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:57.158576    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc" podUID="fe45dacc-3544-4123-a779-db0455a8a702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 29 23:22:57.179484 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:57.179353    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:58.161828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:58.157585    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 29 23:22:58.161828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:58.157663    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc" podUID="fe45dacc-3544-4123-a779-db0455a8a702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 29 23:22:58.439289 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:58.439200    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:22:58.860669 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:58.860567    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"]
Oct 29 23:22:58.881824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:58.877582    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"]
Oct 29 23:22:58.881824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:58.877706    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:58.895997 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:58.894245    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-4ad17e-openshift-default-dockercfg-pkm5r\""
Oct 29 23:22:59.000325 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.000291    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-workload-socket\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.000495 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.000372    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-podinfo\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.000543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.000501    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-token\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.000580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.000540    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf5g5\" (UniqueName: \"kubernetes.io/projected/45b6bf03-44e3-43c8-9ed9-7984e56e031e-kube-api-access-nf5g5\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.000617 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.000598    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-data\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.000659 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.000634    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-credential-socket\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.000696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.000652    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-envoy\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.000779 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.000766    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istiod-ca-cert\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.000880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.000834    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-workload-certs\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.102424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.102386    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-workload-socket\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.102631 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.102462    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-podinfo\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.102631 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.102573    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-token\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.102631 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.102610    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf5g5\" (UniqueName: \"kubernetes.io/projected/45b6bf03-44e3-43c8-9ed9-7984e56e031e-kube-api-access-nf5g5\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.102857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.102646    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-data\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.102857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.102688    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-credential-socket\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.102857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.102712    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-envoy\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.102857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.102845    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-workload-socket\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.103091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.102889    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istiod-ca-cert\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.103091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.102955    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-workload-certs\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.103214 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.103095    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-data\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.103374 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.103353    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-workload-certs\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.103443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.103358    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-credential-socket\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.103841 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.103796    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istiod-ca-cert\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.106094 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.106072    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-envoy\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.109838 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.107567    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-podinfo\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.162824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.161390    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-token\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.164006 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.163977    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf5g5\" (UniqueName: \"kubernetes.io/projected/45b6bf03-44e3-43c8-9ed9-7984e56e031e-kube-api-access-nf5g5\") pod \"e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") " pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.193290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.193221    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:59.193972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.193828    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:59.223039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.220529    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:22:59.223039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.220973    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:22:59.668015 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.667989    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"]
Oct 29 23:22:59.668730 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:22:59.668700    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b6bf03_44e3_43c8_9ed9_7984e56e031e.slice/crio-fbbab6c96feea0ce6aeb0006c87d24fbd470348c2f2feb26c6f5e1c16782050c WatchSource:0}: Error finding container fbbab6c96feea0ce6aeb0006c87d24fbd470348c2f2feb26c6f5e1c16782050c: Status 404 returned error can't find the container with id fbbab6c96feea0ce6aeb0006c87d24fbd470348c2f2feb26c6f5e1c16782050c
Oct 29 23:22:59.671612 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.671581    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:22:59.671680 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.671645    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:22:59.671721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:22:59.671676    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:23:00.557317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:00.557277    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm" event={"ID":"45b6bf03-44e3-43c8-9ed9-7984e56e031e","Type":"ContainerStarted","Data":"f6aa11558873bdedd9ea069b9522067fe4ef1dbd0a60c626b5b07ae72190d746"}
Oct 29 23:23:00.557829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:00.557340    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm" event={"ID":"45b6bf03-44e3-43c8-9ed9-7984e56e031e","Type":"ContainerStarted","Data":"fbbab6c96feea0ce6aeb0006c87d24fbd470348c2f2feb26c6f5e1c16782050c"}
Oct 29 23:23:00.634309 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:00.634255    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm" podStartSLOduration=2.634240316 podStartE2EDuration="2.634240316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:00.618699498 +0000 UTC m=+2383.961648245" watchObservedRunningTime="2026-10-29 23:23:00.634240316 +0000 UTC m=+2383.977189061"
Oct 29 23:23:00.888224 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:00.887950    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:01.224032 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:01.223952    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:23:01.226063 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:01.226037    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 29 23:23:01.226240 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:01.226089    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm" podUID="45b6bf03-44e3-43c8-9ed9-7984e56e031e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 29 23:23:02.224011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:02.223973    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 29 23:23:02.224405 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:02.224036    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm" podUID="45b6bf03-44e3-43c8-9ed9-7984e56e031e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 29 23:23:02.637092 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:02.637062    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:03.224154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:03.224115    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 29 23:23:03.224575 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:03.224180    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm" podUID="45b6bf03-44e3-43c8-9ed9-7984e56e031e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 29 23:23:04.223754 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.223720    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 29 23:23:04.223972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.223787    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm" podUID="45b6bf03-44e3-43c8-9ed9-7984e56e031e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 29 23:23:04.230391 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.230362    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"]
Oct 29 23:23:04.236612 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.236594    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"
Oct 29 23:23:04.253685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.253661    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-6f162a-serving-cert\""
Oct 29 23:23:04.256626 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.256601    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"]
Oct 29 23:23:04.344658 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.344624    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3555bd1b-d45c-4196-953d-af8ed8af9e63-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb\" (UID: \"3555bd1b-d45c-4196-953d-af8ed8af9e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"
Oct 29 23:23:04.344893 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.344869    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbd5p\" (UniqueName: \"kubernetes.io/projected/3555bd1b-d45c-4196-953d-af8ed8af9e63-kube-api-access-rbd5p\") pod \"maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb\" (UID: \"3555bd1b-d45c-4196-953d-af8ed8af9e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"
Oct 29 23:23:04.344973 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.344909    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3555bd1b-d45c-4196-953d-af8ed8af9e63-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb\" (UID: \"3555bd1b-d45c-4196-953d-af8ed8af9e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"
Oct 29 23:23:04.345021 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.344973    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3555bd1b-d45c-4196-953d-af8ed8af9e63-metrics-certs\") pod \"maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb\" (UID: \"3555bd1b-d45c-4196-953d-af8ed8af9e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"
Oct 29 23:23:04.385868 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.385786    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:04.446379 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.446346    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3555bd1b-d45c-4196-953d-af8ed8af9e63-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb\" (UID: \"3555bd1b-d45c-4196-953d-af8ed8af9e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"
Oct 29 23:23:04.446578 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.446502    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbd5p\" (UniqueName: \"kubernetes.io/projected/3555bd1b-d45c-4196-953d-af8ed8af9e63-kube-api-access-rbd5p\") pod \"maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb\" (UID: \"3555bd1b-d45c-4196-953d-af8ed8af9e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"
Oct 29 23:23:04.446578 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.446526    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3555bd1b-d45c-4196-953d-af8ed8af9e63-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb\" (UID: \"3555bd1b-d45c-4196-953d-af8ed8af9e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"
Oct 29 23:23:04.446578 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.446556    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3555bd1b-d45c-4196-953d-af8ed8af9e63-metrics-certs\") pod \"maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb\" (UID: \"3555bd1b-d45c-4196-953d-af8ed8af9e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"
Oct 29 23:23:04.447128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.447103    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3555bd1b-d45c-4196-953d-af8ed8af9e63-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb\" (UID: \"3555bd1b-d45c-4196-953d-af8ed8af9e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"
Oct 29 23:23:04.449143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.449125    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3555bd1b-d45c-4196-953d-af8ed8af9e63-metrics-certs\") pod \"maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb\" (UID: \"3555bd1b-d45c-4196-953d-af8ed8af9e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"
Oct 29 23:23:04.449202 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.449176    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3555bd1b-d45c-4196-953d-af8ed8af9e63-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb\" (UID: \"3555bd1b-d45c-4196-953d-af8ed8af9e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"
Oct 29 23:23:04.498876 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.492955    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbd5p\" (UniqueName: \"kubernetes.io/projected/3555bd1b-d45c-4196-953d-af8ed8af9e63-kube-api-access-rbd5p\") pod \"maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb\" (UID: \"3555bd1b-d45c-4196-953d-af8ed8af9e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"
Oct 29 23:23:04.510158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.509997    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:04.552591 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.552560    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"
Oct 29 23:23:04.921749 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:04.921726    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"]
Oct 29 23:23:04.922713 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:23:04.922688    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3555bd1b_d45c_4196_953d_af8ed8af9e63.slice/crio-d3ca839ed30616103ada2f11fd0b8de643a19302a38593f331ac3bfe189ddab8 WatchSource:0}: Error finding container d3ca839ed30616103ada2f11fd0b8de643a19302a38593f331ac3bfe189ddab8: Status 404 returned error can't find the container with id d3ca839ed30616103ada2f11fd0b8de643a19302a38593f331ac3bfe189ddab8
Oct 29 23:23:05.224040 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:05.223966    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 29 23:23:05.224040 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:05.224015    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm" podUID="45b6bf03-44e3-43c8-9ed9-7984e56e031e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 29 23:23:05.637823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:05.633635    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb" event={"ID":"3555bd1b-d45c-4196-953d-af8ed8af9e63","Type":"ContainerStarted","Data":"7d0715accfeb3b27ef295776f7f45b5fb806db8e8b6fa01934b4a07b6be16efd"}
Oct 29 23:23:05.637823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:05.633704    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb" event={"ID":"3555bd1b-d45c-4196-953d-af8ed8af9e63","Type":"ContainerStarted","Data":"d3ca839ed30616103ada2f11fd0b8de643a19302a38593f331ac3bfe189ddab8"}
Oct 29 23:23:05.637823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:05.634918    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"
Oct 29 23:23:06.228782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:06.228746    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:23:06.229338 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:06.229158    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:23:06.230112 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:06.230093    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:23:06.286459 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:06.286400    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb" podStartSLOduration=1.87613987 podStartE2EDuration="2.286384915s" totalImagesPullingTime="410.245045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:05.688956855 +0000 UTC m=+2389.031905642" watchObservedRunningTime="2026-10-29 23:23:06.286384915 +0000 UTC m=+2389.629333663"
Oct 29 23:23:06.292939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:06.292907    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:08.084544 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:08.084511    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:09.402523 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:09.402488    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"]
Oct 29 23:23:09.403079 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:09.402727    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4" podUID="b2c4718e-8b21-4bee-a55c-22c06f27cd3a" containerName="istio-proxy" containerID="cri-o://7d02d9d7c0a6573f829cb5641e7834346f6076497f602fde1ab519d04b5b96b8" gracePeriod=30
Oct 29 23:23:09.931468 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:09.931437    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:10.055872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:10.055816    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:10.175293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:10.175261    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:11.902596 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:11.902560    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:12.662610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:12.662580    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"
Oct 29 23:23:13.771388 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:13.771350    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:14.700144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.700110    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:23:14.710001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.709967    5428 generic.go:356] "Generic (PLEG): container finished" podID="b2c4718e-8b21-4bee-a55c-22c06f27cd3a" containerID="7d02d9d7c0a6573f829cb5641e7834346f6076497f602fde1ab519d04b5b96b8" exitCode=0
Oct 29 23:23:14.710157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.710014    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"
Oct 29 23:23:14.710157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.710029    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4" event={"ID":"b2c4718e-8b21-4bee-a55c-22c06f27cd3a","Type":"ContainerDied","Data":"7d02d9d7c0a6573f829cb5641e7834346f6076497f602fde1ab519d04b5b96b8"}
Oct 29 23:23:14.710157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.710081    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4" event={"ID":"b2c4718e-8b21-4bee-a55c-22c06f27cd3a","Type":"ContainerDied","Data":"3040bbf32a61d643a4a240d9828a561a3a7eebc059181f35985bd9bd9a57cf65"}
Oct 29 23:23:14.710157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.710112    5428 scope.go:122] "RemoveContainer" containerID="7d02d9d7c0a6573f829cb5641e7834346f6076497f602fde1ab519d04b5b96b8"
Oct 29 23:23:14.721224 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.721206    5428 scope.go:122] "RemoveContainer" containerID="7d02d9d7c0a6573f829cb5641e7834346f6076497f602fde1ab519d04b5b96b8"
Oct 29 23:23:14.721495 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:23:14.721476    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d02d9d7c0a6573f829cb5641e7834346f6076497f602fde1ab519d04b5b96b8\": container with ID starting with 7d02d9d7c0a6573f829cb5641e7834346f6076497f602fde1ab519d04b5b96b8 not found: ID does not exist" containerID="7d02d9d7c0a6573f829cb5641e7834346f6076497f602fde1ab519d04b5b96b8"
Oct 29 23:23:14.721540 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.721502    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d02d9d7c0a6573f829cb5641e7834346f6076497f602fde1ab519d04b5b96b8"} err="failed to get container status \"7d02d9d7c0a6573f829cb5641e7834346f6076497f602fde1ab519d04b5b96b8\": rpc error: code = NotFound desc = could not find container \"7d02d9d7c0a6573f829cb5641e7834346f6076497f602fde1ab519d04b5b96b8\": container with ID starting with 7d02d9d7c0a6573f829cb5641e7834346f6076497f602fde1ab519d04b5b96b8 not found: ID does not exist"
Oct 29 23:23:14.735210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.735186    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istiod-ca-cert\") pod \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") "
Oct 29 23:23:14.735321 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.735277    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-credential-socket\") pod \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") "
Oct 29 23:23:14.735369 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.735358    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-podinfo\") pod \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") "
Oct 29 23:23:14.735413 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.735394    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-workload-certs\") pod \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") "
Oct 29 23:23:14.735451 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.735419    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-kube-api-access-5l9js\" (UniqueName: \"kubernetes.io/projected/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-kube-api-access-5l9js\") pod \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") "
Oct 29 23:23:14.735498 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.735482    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-workload-socket\") pod \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") "
Oct 29 23:23:14.735564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.735545    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-data\") pod \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") "
Oct 29 23:23:14.735633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.735568    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-credential-socket" pod "b2c4718e-8b21-4bee-a55c-22c06f27cd3a" (UID: "b2c4718e-8b21-4bee-a55c-22c06f27cd3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:23:14.735633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.735599    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-envoy\") pod \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") "
Oct 29 23:23:14.735730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.735665    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-workload-certs" pod "b2c4718e-8b21-4bee-a55c-22c06f27cd3a" (UID: "b2c4718e-8b21-4bee-a55c-22c06f27cd3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:23:14.735730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.735678    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-token\" (UniqueName: \"kubernetes.io/projected/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-token\") pod \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\" (UID: \"b2c4718e-8b21-4bee-a55c-22c06f27cd3a\") "
Oct 29 23:23:14.735730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.735665    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istiod-ca-cert" pod "b2c4718e-8b21-4bee-a55c-22c06f27cd3a" (UID: "b2c4718e-8b21-4bee-a55c-22c06f27cd3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:23:14.735922 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.735878    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-data" pod "b2c4718e-8b21-4bee-a55c-22c06f27cd3a" (UID: "b2c4718e-8b21-4bee-a55c-22c06f27cd3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:23:14.735922 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.735897    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-workload-socket" pod "b2c4718e-8b21-4bee-a55c-22c06f27cd3a" (UID: "b2c4718e-8b21-4bee-a55c-22c06f27cd3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:23:14.736355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.736335    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:23:14.736447 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.736359    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:23:14.736447 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.736372    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:23:14.736447 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.736384    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:23:14.736447 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.736397    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:23:14.741266 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.741243    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-envoy" pod "b2c4718e-8b21-4bee-a55c-22c06f27cd3a" (UID: "b2c4718e-8b21-4bee-a55c-22c06f27cd3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:23:14.741387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.741264    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-kube-api-access-5l9js" pod "b2c4718e-8b21-4bee-a55c-22c06f27cd3a" (UID: "b2c4718e-8b21-4bee-a55c-22c06f27cd3a"). InnerVolumeSpecName "kube-api-access-5l9js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:23:14.741387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.741289    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-podinfo" pod "b2c4718e-8b21-4bee-a55c-22c06f27cd3a" (UID: "b2c4718e-8b21-4bee-a55c-22c06f27cd3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:23:14.741484 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.741408    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-token" pod "b2c4718e-8b21-4bee-a55c-22c06f27cd3a" (UID: "b2c4718e-8b21-4bee-a55c-22c06f27cd3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:23:14.838222 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.838188    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:23:14.838222 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.838221    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l9js\" (UniqueName: \"kubernetes.io/projected/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-kube-api-access-5l9js\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:23:14.838613 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.838232    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:23:14.838613 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:14.838243    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2c4718e-8b21-4bee-a55c-22c06f27cd3a-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:23:15.052572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:15.052482    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"]
Oct 29 23:23:15.064531 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:15.064501    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-bb9d4"]
Oct 29 23:23:15.581744 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:15.581695    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:16.780401 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:16.780365    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c4718e-8b21-4bee-a55c-22c06f27cd3a" path="/var/lib/kubelet/pods/b2c4718e-8b21-4bee-a55c-22c06f27cd3a/volumes"
Oct 29 23:23:17.351978 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:17.351948    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:17.371692 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:23:17.371659    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:23:17.482142 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:17.479917    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:19.220006 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:19.219960    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:20.991594 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:20.991559    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:22.758592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:22.758545    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:24.526631 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:24.526599    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:24.664601 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:24.664567    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:24.914668 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:24.914633    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:27.394767 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:27.394728    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:29.633734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:29.633705    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:29.753407 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:29.753369    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:30.222664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:30.222631    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:31.055548 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:31.055509    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:31.559036 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:31.559004    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:31.690668 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:31.690589    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:31.922721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:31.922687    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:33.833754 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:33.833718    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:35.760015 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:35.759979    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:35.874315 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:35.874278    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:36.006834 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:36.006364    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:37.003400 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:37.003370    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:37.408906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:37.408870    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:37.530780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:37.530739    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:38.489227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:38.489194    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:40.304428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:40.304399    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:42.354664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:42.354632    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:42.904892 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:42.904854    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:43.023453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:43.023418    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:43.548188 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:43.548160    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:44.834051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:44.834017    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:46.826506 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:46.826466    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:47.369097 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:23:47.369062    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:23:48.783676 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:48.783643    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:49.898701 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:49.898671    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:50.017907 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:50.017873    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:50.130989 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:50.130958    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:50.711230 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:50.711198    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:52.479183 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:52.479154    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:54.342680 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:54.342648    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:56.087548 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:56.087510    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:56.227858 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:56.226124    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:56.347142 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:56.347063    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:56.583390 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:56.583358    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:56.738755 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:56.738721    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:56.855459 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:56.855422    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:57.063962 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:57.063868    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:57.684472 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:57.684354    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:57.800007 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:57.799975    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:58.512043 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:58.512010    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:23:58.931595 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:23:58.931549    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:00.525346 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:00.525310    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:02.303528 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:02.303496    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:04.238065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:04.238025    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:06.195996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:06.195962    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:08.009213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:08.009180    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:08.692515 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:08.692480    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:10.004602 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:10.004565    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:10.121700 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:10.121666    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:10.582334 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:10.582302    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:10.740343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:10.740307    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:12.239246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:12.239209    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:14.223558 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:14.223523    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:14.492275 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:14.492203    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:14.613254 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:14.613217    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:14.877788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:14.877760    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:16.096487 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.096453    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:16.207489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.207458    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:16.269860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.269828    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd"]
Oct 29 23:24:16.270754 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.270736    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b2c4718e-8b21-4bee-a55c-22c06f27cd3a" containerName="istio-proxy"
Oct 29 23:24:16.270841 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.270755    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c4718e-8b21-4bee-a55c-22c06f27cd3a" containerName="istio-proxy"
Oct 29 23:24:16.270935 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.270925    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c4718e-8b21-4bee-a55c-22c06f27cd3a" containerName="istio-proxy"
Oct 29 23:24:16.275152 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.275130    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd"
Oct 29 23:24:16.294049 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.294023    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd"]
Oct 29 23:24:16.300714 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.300686    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-fb367d\""
Oct 29 23:24:16.300947 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.300820    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-fb367d-dockercfg-456xb\""
Oct 29 23:24:16.329281 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.329253    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:16.339549 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.339525    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8"]
Oct 29 23:24:16.343721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.343698    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df13b19-5a4b-4eff-8733-66c06fdb38eb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd\" (UID: \"9df13b19-5a4b-4eff-8733-66c06fdb38eb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd"
Oct 29 23:24:16.343883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.343868    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt4ql\" (UniqueName: \"kubernetes.io/projected/9df13b19-5a4b-4eff-8733-66c06fdb38eb-kube-api-access-gt4ql\") pod \"payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd\" (UID: \"9df13b19-5a4b-4eff-8733-66c06fdb38eb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd"
Oct 29 23:24:16.344565 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.344550    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8"
Oct 29 23:24:16.365909 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.365875    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8"]
Oct 29 23:24:16.491825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.457483    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c1689ec-01d8-4ef6-885c-e15f332d06a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8\" (UID: \"4c1689ec-01d8-4ef6-885c-e15f332d06a0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8"
Oct 29 23:24:16.520824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.518021    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt4ql\" (UniqueName: \"kubernetes.io/projected/9df13b19-5a4b-4eff-8733-66c06fdb38eb-kube-api-access-gt4ql\") pod \"payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd\" (UID: \"9df13b19-5a4b-4eff-8733-66c06fdb38eb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd"
Oct 29 23:24:16.520824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.518197    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49k7c\" (UniqueName: \"kubernetes.io/projected/4c1689ec-01d8-4ef6-885c-e15f332d06a0-kube-api-access-49k7c\") pod \"payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8\" (UID: \"4c1689ec-01d8-4ef6-885c-e15f332d06a0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8"
Oct 29 23:24:16.520824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.518430    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df13b19-5a4b-4eff-8733-66c06fdb38eb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd\" (UID: \"9df13b19-5a4b-4eff-8733-66c06fdb38eb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd"
Oct 29 23:24:16.520824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.519456    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df13b19-5a4b-4eff-8733-66c06fdb38eb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd\" (UID: \"9df13b19-5a4b-4eff-8733-66c06fdb38eb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd"
Oct 29 23:24:16.579612 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.579544    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt4ql\" (UniqueName: \"kubernetes.io/projected/9df13b19-5a4b-4eff-8733-66c06fdb38eb-kube-api-access-gt4ql\") pod \"payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd\" (UID: \"9df13b19-5a4b-4eff-8733-66c06fdb38eb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd"
Oct 29 23:24:16.593774 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.593744    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd"
Oct 29 23:24:16.609514 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.609441    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:16.620582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.620552    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c1689ec-01d8-4ef6-885c-e15f332d06a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8\" (UID: \"4c1689ec-01d8-4ef6-885c-e15f332d06a0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8"
Oct 29 23:24:16.620774 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.620753    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49k7c\" (UniqueName: \"kubernetes.io/projected/4c1689ec-01d8-4ef6-885c-e15f332d06a0-kube-api-access-49k7c\") pod \"payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8\" (UID: \"4c1689ec-01d8-4ef6-885c-e15f332d06a0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8"
Oct 29 23:24:16.621363 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.621342    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c1689ec-01d8-4ef6-885c-e15f332d06a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8\" (UID: \"4c1689ec-01d8-4ef6-885c-e15f332d06a0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8"
Oct 29 23:24:16.645498 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.645464    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49k7c\" (UniqueName: \"kubernetes.io/projected/4c1689ec-01d8-4ef6-885c-e15f332d06a0-kube-api-access-49k7c\") pod \"payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8\" (UID: \"4c1689ec-01d8-4ef6-885c-e15f332d06a0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8"
Oct 29 23:24:16.660116 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.660078    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8"
Oct 29 23:24:16.909330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:16.909305    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd"]
Oct 29 23:24:16.909910 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:24:16.909888    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df13b19_5a4b_4eff_8733_66c06fdb38eb.slice/crio-6634a13d35e2be0efe8c02a49fab11936078bbd1d85d898978908407d2324d54 WatchSource:0}: Error finding container 6634a13d35e2be0efe8c02a49fab11936078bbd1d85d898978908407d2324d54: Status 404 returned error can't find the container with id 6634a13d35e2be0efe8c02a49fab11936078bbd1d85d898978908407d2324d54
Oct 29 23:24:17.036563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:17.036538    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8"]
Oct 29 23:24:17.038307 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:24:17.038277    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1689ec_01d8_4ef6_885c_e15f332d06a0.slice/crio-dd0a657adeed6201bb2dc9cb5e96024e6faad7fbd08fcac41de069cefa6addc8 WatchSource:0}: Error finding container dd0a657adeed6201bb2dc9cb5e96024e6faad7fbd08fcac41de069cefa6addc8: Status 404 returned error can't find the container with id dd0a657adeed6201bb2dc9cb5e96024e6faad7fbd08fcac41de069cefa6addc8
Oct 29 23:24:17.273160 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:17.273082    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd" event={"ID":"9df13b19-5a4b-4eff-8733-66c06fdb38eb","Type":"ContainerStarted","Data":"d35b7ebb92466f11b1884dc10ddacb4795175a42139fb8d917888052c8d5c46b"}
Oct 29 23:24:17.273160 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:17.273134    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd" event={"ID":"9df13b19-5a4b-4eff-8733-66c06fdb38eb","Type":"ContainerStarted","Data":"6634a13d35e2be0efe8c02a49fab11936078bbd1d85d898978908407d2324d54"}
Oct 29 23:24:17.273619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:17.273182    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd"
Oct 29 23:24:17.274759 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:17.274738    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8" event={"ID":"4c1689ec-01d8-4ef6-885c-e15f332d06a0","Type":"ContainerStarted","Data":"1b6101dfb0177128d32818cfd80946618dfc19dc83bf7223ec7f89d3e317b70c"}
Oct 29 23:24:17.274842 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:17.274782    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8" event={"ID":"4c1689ec-01d8-4ef6-885c-e15f332d06a0","Type":"ContainerStarted","Data":"dd0a657adeed6201bb2dc9cb5e96024e6faad7fbd08fcac41de069cefa6addc8"}
Oct 29 23:24:17.274889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:17.274847    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8"
Oct 29 23:24:17.317514 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:17.317458    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd" podStartSLOduration=1.317441835 podStartE2EDuration="1.317441835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:17.313103016 +0000 UTC m=+2460.656051789" watchObservedRunningTime="2026-10-29 23:24:17.317441835 +0000 UTC m=+2460.660390581"
Oct 29 23:24:17.353197 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:17.353083    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8" podStartSLOduration=1.353067376 podStartE2EDuration="1.353067376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:17.352866979 +0000 UTC m=+2460.695815724" watchObservedRunningTime="2026-10-29 23:24:17.353067376 +0000 UTC m=+2460.696016123"
Oct 29 23:24:17.370872 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:24:17.370838    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:24:18.512792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:18.512761    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:20.509410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:20.509377    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:20.793146 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:20.793047    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:22.122708 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:22.122675    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:22.525655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:22.525573    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:22.656159 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:22.656127    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:22.796883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:22.796782    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:23.705129 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:23.705081    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:23.820239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:23.820094    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:24.799474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:24.799438    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:26.391537 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:26.391500    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:26.515628 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:26.515599    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:26.641904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:26.641810    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:27.282646 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:27.282617    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:27.839474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:27.839441    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:28.327747 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:28.327717    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd"
Oct 29 23:24:28.327959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:28.327812    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8"
Oct 29 23:24:28.456408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:28.456371    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:28.582375 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:28.582287    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:28.625273 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:28.625238    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr"]
Oct 29 23:24:28.634619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:28.634591    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr"
Oct 29 23:24:28.646506 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:28.646476    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr"]
Oct 29 23:24:28.649346 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:28.649312    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-7d4110\""
Oct 29 23:24:28.649498 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:28.649446    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-7d4110-dockercfg-fdkv4\""
Oct 29 23:24:28.900094 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:28.900051    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ed6510-aaf7-4059-9f80-8cf1595e9c89-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr\" (UID: \"03ed6510-aaf7-4059-9f80-8cf1595e9c89\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr"
Oct 29 23:24:28.900509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:28.900119    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkqhr\" (UniqueName: \"kubernetes.io/projected/03ed6510-aaf7-4059-9f80-8cf1595e9c89-kube-api-access-zkqhr\") pod \"payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr\" (UID: \"03ed6510-aaf7-4059-9f80-8cf1595e9c89\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr"
Oct 29 23:24:29.001405 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:29.001365    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ed6510-aaf7-4059-9f80-8cf1595e9c89-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr\" (UID: \"03ed6510-aaf7-4059-9f80-8cf1595e9c89\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr"
Oct 29 23:24:29.001405 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:29.001412    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqhr\" (UniqueName: \"kubernetes.io/projected/03ed6510-aaf7-4059-9f80-8cf1595e9c89-kube-api-access-zkqhr\") pod \"payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr\" (UID: \"03ed6510-aaf7-4059-9f80-8cf1595e9c89\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr"
Oct 29 23:24:29.002174 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:29.002151    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ed6510-aaf7-4059-9f80-8cf1595e9c89-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr\" (UID: \"03ed6510-aaf7-4059-9f80-8cf1595e9c89\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr"
Oct 29 23:24:29.025038 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:29.025002    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkqhr\" (UniqueName: \"kubernetes.io/projected/03ed6510-aaf7-4059-9f80-8cf1595e9c89-kube-api-access-zkqhr\") pod \"payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr\" (UID: \"03ed6510-aaf7-4059-9f80-8cf1595e9c89\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr"
Oct 29 23:24:29.217358 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:29.217280    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:29.254495 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:29.254470    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr"
Oct 29 23:24:29.564393 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:29.564354    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr"]
Oct 29 23:24:29.564713 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:24:29.564687    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ed6510_aaf7_4059_9f80_8cf1595e9c89.slice/crio-75c74ea8b20dbfd44e38c50985cbec0336c175c713cd87b21a93517264a427b5 WatchSource:0}: Error finding container 75c74ea8b20dbfd44e38c50985cbec0336c175c713cd87b21a93517264a427b5: Status 404 returned error can't find the container with id 75c74ea8b20dbfd44e38c50985cbec0336c175c713cd87b21a93517264a427b5
Oct 29 23:24:29.717682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:29.717645    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:29.813686 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:29.813589    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"]
Oct 29 23:24:29.838076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:29.837999    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf" podUID="2e3ed106-1a9a-43e6-998f-52b37686da76" containerName="istio-proxy" containerID="cri-o://b0f94ed9cf5def08600d9b4815074b7c0d3e700d583ecf77c00592361c34b099" gracePeriod=30
Oct 29 23:24:29.955851 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:29.949697    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:30.093773 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:30.093691    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:30.434543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:30.434508    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr" event={"ID":"03ed6510-aaf7-4059-9f80-8cf1595e9c89","Type":"ContainerStarted","Data":"e8df1e85e46f3e2a09e9dc9c2a67f13ed74def5e7e36432f09849007ac2f4319"}
Oct 29 23:24:30.434735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:30.434561    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr" event={"ID":"03ed6510-aaf7-4059-9f80-8cf1595e9c89","Type":"ContainerStarted","Data":"75c74ea8b20dbfd44e38c50985cbec0336c175c713cd87b21a93517264a427b5"}
Oct 29 23:24:30.434735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:30.434701    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr"
Oct 29 23:24:30.475657 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:30.475607    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr" podStartSLOduration=2.475589662 podStartE2EDuration="2.475589662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:30.474845546 +0000 UTC m=+2473.817794292" watchObservedRunningTime="2026-10-29 23:24:30.475589662 +0000 UTC m=+2473.818538406"
Oct 29 23:24:30.571914 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:30.571885    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:32.317092 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:32.317060    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:34.423405 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:34.423366    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:34.964225 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:34.964189    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:35.082915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.082888    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:35.266481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.266425    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:24:35.325079 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.325047    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-envoy\") pod \"2e3ed106-1a9a-43e6-998f-52b37686da76\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") "
Oct 29 23:24:35.325314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.325177    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-data\") pod \"2e3ed106-1a9a-43e6-998f-52b37686da76\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") "
Oct 29 23:24:35.325314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.325220    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-credential-socket\") pod \"2e3ed106-1a9a-43e6-998f-52b37686da76\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") "
Oct 29 23:24:35.325314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.325252    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e3ed106-1a9a-43e6-998f-52b37686da76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3ed106-1a9a-43e6-998f-52b37686da76-istiod-ca-cert\") pod \"2e3ed106-1a9a-43e6-998f-52b37686da76\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") "
Oct 29 23:24:35.326158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.325389    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-workload-socket\") pod \"2e3ed106-1a9a-43e6-998f-52b37686da76\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") "
Oct 29 23:24:35.326158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.325428    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-token\" (UniqueName: \"kubernetes.io/projected/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-token\") pod \"2e3ed106-1a9a-43e6-998f-52b37686da76\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") "
Oct 29 23:24:35.326158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.325463    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-podinfo\") pod \"2e3ed106-1a9a-43e6-998f-52b37686da76\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") "
Oct 29 23:24:35.326158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.325598    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3ed106-1a9a-43e6-998f-52b37686da76-kube-api-access-8zvmg\" (UniqueName: \"kubernetes.io/projected/2e3ed106-1a9a-43e6-998f-52b37686da76-kube-api-access-8zvmg\") pod \"2e3ed106-1a9a-43e6-998f-52b37686da76\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") "
Oct 29 23:24:35.326158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.325639    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-workload-socket" pod "2e3ed106-1a9a-43e6-998f-52b37686da76" (UID: "2e3ed106-1a9a-43e6-998f-52b37686da76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:35.326158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.325677    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-workload-certs\") pod \"2e3ed106-1a9a-43e6-998f-52b37686da76\" (UID: \"2e3ed106-1a9a-43e6-998f-52b37686da76\") "
Oct 29 23:24:35.326158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.325897    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-credential-socket" pod "2e3ed106-1a9a-43e6-998f-52b37686da76" (UID: "2e3ed106-1a9a-43e6-998f-52b37686da76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:35.327780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.326287    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3ed106-1a9a-43e6-998f-52b37686da76-istiod-ca-cert" pod "2e3ed106-1a9a-43e6-998f-52b37686da76" (UID: "2e3ed106-1a9a-43e6-998f-52b37686da76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:24:35.327780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.326468    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-workload-certs" pod "2e3ed106-1a9a-43e6-998f-52b37686da76" (UID: "2e3ed106-1a9a-43e6-998f-52b37686da76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:35.327780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.326558    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:35.327780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.326573    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3ed106-1a9a-43e6-998f-52b37686da76-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:35.327780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.326582    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:35.327780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.326703    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-data" pod "2e3ed106-1a9a-43e6-998f-52b37686da76" (UID: "2e3ed106-1a9a-43e6-998f-52b37686da76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:35.330464 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.329191    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-podinfo" pod "2e3ed106-1a9a-43e6-998f-52b37686da76" (UID: "2e3ed106-1a9a-43e6-998f-52b37686da76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:24:35.330464 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.329479    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-envoy" pod "2e3ed106-1a9a-43e6-998f-52b37686da76" (UID: "2e3ed106-1a9a-43e6-998f-52b37686da76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:35.330464 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.329598    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3ed106-1a9a-43e6-998f-52b37686da76-kube-api-access-8zvmg" pod "2e3ed106-1a9a-43e6-998f-52b37686da76" (UID: "2e3ed106-1a9a-43e6-998f-52b37686da76"). InnerVolumeSpecName "kube-api-access-8zvmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:35.333101 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.333071    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-token" pod "2e3ed106-1a9a-43e6-998f-52b37686da76" (UID: "2e3ed106-1a9a-43e6-998f-52b37686da76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:35.427640 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.427609    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:35.427640 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.427638    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:35.427640 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.427647    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:35.428072 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.427655    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:35.428072 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.427664    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3ed106-1a9a-43e6-998f-52b37686da76-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:35.428072 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.427673    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zvmg\" (UniqueName: \"kubernetes.io/projected/2e3ed106-1a9a-43e6-998f-52b37686da76-kube-api-access-8zvmg\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:35.481838 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.481735    5428 generic.go:356] "Generic (PLEG): container finished" podID="2e3ed106-1a9a-43e6-998f-52b37686da76" containerID="b0f94ed9cf5def08600d9b4815074b7c0d3e700d583ecf77c00592361c34b099" exitCode=0
Oct 29 23:24:35.481838 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.481779    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"
Oct 29 23:24:35.482019 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.481835    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf" event={"ID":"2e3ed106-1a9a-43e6-998f-52b37686da76","Type":"ContainerDied","Data":"b0f94ed9cf5def08600d9b4815074b7c0d3e700d583ecf77c00592361c34b099"}
Oct 29 23:24:35.482019 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.481891    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf" event={"ID":"2e3ed106-1a9a-43e6-998f-52b37686da76","Type":"ContainerDied","Data":"4a7d9b840a24e753d51de461d9e93dd3ca371664783cde7adbdee8cfc89acf1b"}
Oct 29 23:24:35.482019 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.481916    5428 scope.go:122] "RemoveContainer" containerID="b0f94ed9cf5def08600d9b4815074b7c0d3e700d583ecf77c00592361c34b099"
Oct 29 23:24:35.494024 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.494007    5428 scope.go:122] "RemoveContainer" containerID="b0f94ed9cf5def08600d9b4815074b7c0d3e700d583ecf77c00592361c34b099"
Oct 29 23:24:35.494296 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:24:35.494277    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f94ed9cf5def08600d9b4815074b7c0d3e700d583ecf77c00592361c34b099\": container with ID starting with b0f94ed9cf5def08600d9b4815074b7c0d3e700d583ecf77c00592361c34b099 not found: ID does not exist" containerID="b0f94ed9cf5def08600d9b4815074b7c0d3e700d583ecf77c00592361c34b099"
Oct 29 23:24:35.494339 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.494304    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f94ed9cf5def08600d9b4815074b7c0d3e700d583ecf77c00592361c34b099"} err="failed to get container status \"b0f94ed9cf5def08600d9b4815074b7c0d3e700d583ecf77c00592361c34b099\": rpc error: code = NotFound desc = could not find container \"b0f94ed9cf5def08600d9b4815074b7c0d3e700d583ecf77c00592361c34b099\": container with ID starting with b0f94ed9cf5def08600d9b4815074b7c0d3e700d583ecf77c00592361c34b099 not found: ID does not exist"
Oct 29 23:24:35.535947 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.535921    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"]
Oct 29 23:24:35.543037 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:35.543006    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-j75tf"]
Oct 29 23:24:36.300610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.300581    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"]
Oct 29 23:24:36.301451 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.301436    5428 cpu_manager.go:414] "containerMap: removing container" podUID="2e3ed106-1a9a-43e6-998f-52b37686da76" containerName="istio-proxy"
Oct 29 23:24:36.301509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.301452    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3ed106-1a9a-43e6-998f-52b37686da76" containerName="istio-proxy"
Oct 29 23:24:36.301603 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.301592    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3ed106-1a9a-43e6-998f-52b37686da76" containerName="istio-proxy"
Oct 29 23:24:36.307351 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.307334    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"
Oct 29 23:24:36.322879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.322831    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-4e2ac7-serving-cert\""
Oct 29 23:24:36.342625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.342596    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"]
Oct 29 23:24:36.440852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.440817    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:36.442341 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.442316    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1067f9-ce81-450a-9a8b-8603f7866708-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz\" (UID: \"9e1067f9-ce81-450a-9a8b-8603f7866708\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"
Oct 29 23:24:36.442436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.442350    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1067f9-ce81-450a-9a8b-8603f7866708-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz\" (UID: \"9e1067f9-ce81-450a-9a8b-8603f7866708\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"
Oct 29 23:24:36.442523 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.442503    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e1067f9-ce81-450a-9a8b-8603f7866708-metrics-certs\") pod \"maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz\" (UID: \"9e1067f9-ce81-450a-9a8b-8603f7866708\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"
Oct 29 23:24:36.442559 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.442547    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htzs5\" (UniqueName: \"kubernetes.io/projected/9e1067f9-ce81-450a-9a8b-8603f7866708-kube-api-access-htzs5\") pod \"maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz\" (UID: \"9e1067f9-ce81-450a-9a8b-8603f7866708\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"
Oct 29 23:24:36.543776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.543746    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1067f9-ce81-450a-9a8b-8603f7866708-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz\" (UID: \"9e1067f9-ce81-450a-9a8b-8603f7866708\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"
Oct 29 23:24:36.543974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.543780    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1067f9-ce81-450a-9a8b-8603f7866708-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz\" (UID: \"9e1067f9-ce81-450a-9a8b-8603f7866708\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"
Oct 29 23:24:36.543974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.543922    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e1067f9-ce81-450a-9a8b-8603f7866708-metrics-certs\") pod \"maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz\" (UID: \"9e1067f9-ce81-450a-9a8b-8603f7866708\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"
Oct 29 23:24:36.543974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.543954    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htzs5\" (UniqueName: \"kubernetes.io/projected/9e1067f9-ce81-450a-9a8b-8603f7866708-kube-api-access-htzs5\") pod \"maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz\" (UID: \"9e1067f9-ce81-450a-9a8b-8603f7866708\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"
Oct 29 23:24:36.544562 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.544534    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1067f9-ce81-450a-9a8b-8603f7866708-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz\" (UID: \"9e1067f9-ce81-450a-9a8b-8603f7866708\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"
Oct 29 23:24:36.546385 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.546355    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1067f9-ce81-450a-9a8b-8603f7866708-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz\" (UID: \"9e1067f9-ce81-450a-9a8b-8603f7866708\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"
Oct 29 23:24:36.546385 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.546374    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e1067f9-ce81-450a-9a8b-8603f7866708-metrics-certs\") pod \"maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz\" (UID: \"9e1067f9-ce81-450a-9a8b-8603f7866708\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"
Oct 29 23:24:36.556520 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.556471    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:36.566812 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.566778    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htzs5\" (UniqueName: \"kubernetes.io/projected/9e1067f9-ce81-450a-9a8b-8603f7866708-kube-api-access-htzs5\") pod \"maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz\" (UID: \"9e1067f9-ce81-450a-9a8b-8603f7866708\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"
Oct 29 23:24:36.625612 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.624368    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"
Oct 29 23:24:36.680271 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.679737    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:36.782031 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.781966    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e3ed106-1a9a-43e6-998f-52b37686da76" path="/var/lib/kubelet/pods/2e3ed106-1a9a-43e6-998f-52b37686da76/volumes"
Oct 29 23:24:36.974145 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:36.974114    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"]
Oct 29 23:24:36.975080 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:24:36.975054    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1067f9_ce81_450a_9a8b_8603f7866708.slice/crio-02afc7204ba925eeccaf689a1b6a4c3ab640f0fe806d616d1ef44a7bb11f6418 WatchSource:0}: Error finding container 02afc7204ba925eeccaf689a1b6a4c3ab640f0fe806d616d1ef44a7bb11f6418: Status 404 returned error can't find the container with id 02afc7204ba925eeccaf689a1b6a4c3ab640f0fe806d616d1ef44a7bb11f6418
Oct 29 23:24:37.228786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:37.228699    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:37.378118 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:37.378089    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:37.413410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:37.413386    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt"]
Oct 29 23:24:37.420050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:37.420032    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt"
Oct 29 23:24:37.435726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:37.435703    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d5e482\""
Oct 29 23:24:37.437794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:37.437772    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt"]
Oct 29 23:24:37.492155 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:37.479641    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d5e482-dockercfg-x5488\""
Oct 29 23:24:37.555866 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:37.548514    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz" event={"ID":"9e1067f9-ce81-450a-9a8b-8603f7866708","Type":"ContainerStarted","Data":"63626851e753e6b0ecf631602800d5c284ecfafb72c133e596b0a79b56126e42"}
Oct 29 23:24:37.555866 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:37.548583    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz" event={"ID":"9e1067f9-ce81-450a-9a8b-8603f7866708","Type":"ContainerStarted","Data":"02afc7204ba925eeccaf689a1b6a4c3ab640f0fe806d616d1ef44a7bb11f6418"}
Oct 29 23:24:37.555866 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:37.549757    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"
Oct 29 23:24:37.569861 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:37.567919    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26sh2\" (UniqueName: \"kubernetes.io/projected/7bb46ed8-8d72-48c9-baa5-543d8a1824fb-kube-api-access-26sh2\") pod \"payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt\" (UID: \"7bb46ed8-8d72-48c9-baa5-543d8a1824fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt"
Oct 29 23:24:37.569861 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:37.568026    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb46ed8-8d72-48c9-baa5-543d8a1824fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt\" (UID: \"7bb46ed8-8d72-48c9-baa5-543d8a1824fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt"
Oct 29 23:24:37.598972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:37.598916    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz" podStartSLOduration=1.177037621 podStartE2EDuration="1.598897188s" totalImagesPullingTime="421.859567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:37.596621724 +0000 UTC m=+2480.939570470" watchObservedRunningTime="2026-10-29 23:24:37.598897188 +0000 UTC m=+2480.941845952"
Oct 29 23:24:37.676524 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:37.673454    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26sh2\" (UniqueName: \"kubernetes.io/projected/7bb46ed8-8d72-48c9-baa5-543d8a1824fb-kube-api-access-26sh2\") pod \"payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt\" (UID: \"7bb46ed8-8d72-48c9-baa5-543d8a1824fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt"
Oct 29 23:24:37.676524 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:37.673548    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb46ed8-8d72-48c9-baa5-543d8a1824fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt\" (UID: \"7bb46ed8-8d72-48c9-baa5-543d8a1824fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt"
Oct 29 23:24:37.676524 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:37.674770    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb46ed8-8d72-48c9-baa5-543d8a1824fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt\" (UID: \"7bb46ed8-8d72-48c9-baa5-543d8a1824fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt"
Oct 29 23:24:37.697142 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:37.697106    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26sh2\" (UniqueName: \"kubernetes.io/projected/7bb46ed8-8d72-48c9-baa5-543d8a1824fb-kube-api-access-26sh2\") pod \"payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt\" (UID: \"7bb46ed8-8d72-48c9-baa5-543d8a1824fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt"
Oct 29 23:24:37.755487 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:37.755414    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt"
Oct 29 23:24:38.086080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:38.086003    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt"]
Oct 29 23:24:38.091105 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:24:38.091076    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb46ed8_8d72_48c9_baa5_543d8a1824fb.slice/crio-0342ee334d1314f749bdb5e0fe1f967fec8072d78e3a8023c920d9a69d58211a WatchSource:0}: Error finding container 0342ee334d1314f749bdb5e0fe1f967fec8072d78e3a8023c920d9a69d58211a: Status 404 returned error can't find the container with id 0342ee334d1314f749bdb5e0fe1f967fec8072d78e3a8023c920d9a69d58211a
Oct 29 23:24:38.134193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:38.134163    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"]
Oct 29 23:24:38.135114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:38.134410    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr" podUID="1a967af1-6e08-4983-9493-0292d3ea8aaa" containerName="istio-proxy" containerID="cri-o://db51ae4c11874fe0170c333811bfaadd220dc85b374e30b35b3a4522d9563fe5" gracePeriod=30
Oct 29 23:24:38.560879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:38.560839    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt" event={"ID":"7bb46ed8-8d72-48c9-baa5-543d8a1824fb","Type":"ContainerStarted","Data":"039052ba8a003a1945e28bade27f22594c9459d2335acb7f9b5de54d1d990620"}
Oct 29 23:24:38.561284 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:38.560908    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt" event={"ID":"7bb46ed8-8d72-48c9-baa5-543d8a1824fb","Type":"ContainerStarted","Data":"0342ee334d1314f749bdb5e0fe1f967fec8072d78e3a8023c920d9a69d58211a"}
Oct 29 23:24:38.561284 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:38.560943    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt"
Oct 29 23:24:38.604711 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:38.604661    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt" podStartSLOduration=1.604646434 podStartE2EDuration="1.604646434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:38.600489682 +0000 UTC m=+2481.943438427" watchObservedRunningTime="2026-10-29 23:24:38.604646434 +0000 UTC m=+2481.947595178"
Oct 29 23:24:38.793705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:38.793674    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:40.869087 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:40.869053    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:41.157335 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:41.157241    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:41.446132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:41.446055    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr"
Oct 29 23:24:41.865074 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:41.865045    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"]
Oct 29 23:24:41.865275 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:41.865234    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd" podUID="ae1fd856-9cb0-4d65-bc25-49aefe12e84a" containerName="istio-proxy" containerID="cri-o://c783e485126991a6cb5dc6acbebe6b1e208c9377dd1b110e7e90a1cf8862e358" gracePeriod=30
Oct 29 23:24:43.155419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.155387    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:43.431746 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.431722    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:24:43.456945 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.456911    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-podinfo\") pod \"1a967af1-6e08-4983-9493-0292d3ea8aaa\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") "
Oct 29 23:24:43.457077 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.457038    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-workload-certs\") pod \"1a967af1-6e08-4983-9493-0292d3ea8aaa\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") "
Oct 29 23:24:43.457077 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.457071    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-envoy\") pod \"1a967af1-6e08-4983-9493-0292d3ea8aaa\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") "
Oct 29 23:24:43.457179 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.457162    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a967af1-6e08-4983-9493-0292d3ea8aaa-kube-api-access-tt2j7\" (UniqueName: \"kubernetes.io/projected/1a967af1-6e08-4983-9493-0292d3ea8aaa-kube-api-access-tt2j7\") pod \"1a967af1-6e08-4983-9493-0292d3ea8aaa\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") "
Oct 29 23:24:43.457269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.457250    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-token\" (UniqueName: \"kubernetes.io/projected/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-token\") pod \"1a967af1-6e08-4983-9493-0292d3ea8aaa\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") "
Oct 29 23:24:43.457425 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.457321    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a967af1-6e08-4983-9493-0292d3ea8aaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a967af1-6e08-4983-9493-0292d3ea8aaa-istiod-ca-cert\") pod \"1a967af1-6e08-4983-9493-0292d3ea8aaa\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") "
Oct 29 23:24:43.457425 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.457352    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-workload-certs" pod "1a967af1-6e08-4983-9493-0292d3ea8aaa" (UID: "1a967af1-6e08-4983-9493-0292d3ea8aaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:43.457559 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.457427    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-credential-socket\") pod \"1a967af1-6e08-4983-9493-0292d3ea8aaa\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") "
Oct 29 23:24:43.457559 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.457460    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-workload-socket\") pod \"1a967af1-6e08-4983-9493-0292d3ea8aaa\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") "
Oct 29 23:24:43.457559 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.457507    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-data\") pod \"1a967af1-6e08-4983-9493-0292d3ea8aaa\" (UID: \"1a967af1-6e08-4983-9493-0292d3ea8aaa\") "
Oct 29 23:24:43.457835 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.457796    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-workload-socket" pod "1a967af1-6e08-4983-9493-0292d3ea8aaa" (UID: "1a967af1-6e08-4983-9493-0292d3ea8aaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:43.457945 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.457829    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a967af1-6e08-4983-9493-0292d3ea8aaa-istiod-ca-cert" pod "1a967af1-6e08-4983-9493-0292d3ea8aaa" (UID: "1a967af1-6e08-4983-9493-0292d3ea8aaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:24:43.457945 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.457870    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-credential-socket" pod "1a967af1-6e08-4983-9493-0292d3ea8aaa" (UID: "1a967af1-6e08-4983-9493-0292d3ea8aaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:43.458125 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.458010    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-data" pod "1a967af1-6e08-4983-9493-0292d3ea8aaa" (UID: "1a967af1-6e08-4983-9493-0292d3ea8aaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:43.458248 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.458227    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a967af1-6e08-4983-9493-0292d3ea8aaa-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:43.458287 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.458255    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:43.458287 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.458268    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:43.458287 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.458281    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:43.458382 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.458292    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:43.459840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.459814    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-envoy" pod "1a967af1-6e08-4983-9493-0292d3ea8aaa" (UID: "1a967af1-6e08-4983-9493-0292d3ea8aaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:43.459937 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.459865    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a967af1-6e08-4983-9493-0292d3ea8aaa-kube-api-access-tt2j7" pod "1a967af1-6e08-4983-9493-0292d3ea8aaa" (UID: "1a967af1-6e08-4983-9493-0292d3ea8aaa"). InnerVolumeSpecName "kube-api-access-tt2j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:43.460009 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.459928    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-podinfo" pod "1a967af1-6e08-4983-9493-0292d3ea8aaa" (UID: "1a967af1-6e08-4983-9493-0292d3ea8aaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:24:43.460049 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.460031    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-token" pod "1a967af1-6e08-4983-9493-0292d3ea8aaa" (UID: "1a967af1-6e08-4983-9493-0292d3ea8aaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:43.559739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.559698    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:43.559739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.559734    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt2j7\" (UniqueName: \"kubernetes.io/projected/1a967af1-6e08-4983-9493-0292d3ea8aaa-kube-api-access-tt2j7\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:43.559739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.559749    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:43.559997 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.559761    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a967af1-6e08-4983-9493-0292d3ea8aaa-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:43.606780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.606746    5428 generic.go:356] "Generic (PLEG): container finished" podID="1a967af1-6e08-4983-9493-0292d3ea8aaa" containerID="db51ae4c11874fe0170c333811bfaadd220dc85b374e30b35b3a4522d9563fe5" exitCode=0
Oct 29 23:24:43.607166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.606838    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"
Oct 29 23:24:43.607341 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.606861    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr" event={"ID":"1a967af1-6e08-4983-9493-0292d3ea8aaa","Type":"ContainerDied","Data":"db51ae4c11874fe0170c333811bfaadd220dc85b374e30b35b3a4522d9563fe5"}
Oct 29 23:24:43.607465 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.607444    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr" event={"ID":"1a967af1-6e08-4983-9493-0292d3ea8aaa","Type":"ContainerDied","Data":"e64d6efb4bd9eaa0833aad9a1515c0731e62148448553ab85454116f211c2370"}
Oct 29 23:24:43.607528 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.607492    5428 scope.go:122] "RemoveContainer" containerID="db51ae4c11874fe0170c333811bfaadd220dc85b374e30b35b3a4522d9563fe5"
Oct 29 23:24:43.620660 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.620643    5428 scope.go:122] "RemoveContainer" containerID="db51ae4c11874fe0170c333811bfaadd220dc85b374e30b35b3a4522d9563fe5"
Oct 29 23:24:43.620976 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:24:43.620958    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db51ae4c11874fe0170c333811bfaadd220dc85b374e30b35b3a4522d9563fe5\": container with ID starting with db51ae4c11874fe0170c333811bfaadd220dc85b374e30b35b3a4522d9563fe5 not found: ID does not exist" containerID="db51ae4c11874fe0170c333811bfaadd220dc85b374e30b35b3a4522d9563fe5"
Oct 29 23:24:43.621037 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.620985    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db51ae4c11874fe0170c333811bfaadd220dc85b374e30b35b3a4522d9563fe5"} err="failed to get container status \"db51ae4c11874fe0170c333811bfaadd220dc85b374e30b35b3a4522d9563fe5\": rpc error: code = NotFound desc = could not find container \"db51ae4c11874fe0170c333811bfaadd220dc85b374e30b35b3a4522d9563fe5\": container with ID starting with db51ae4c11874fe0170c333811bfaadd220dc85b374e30b35b3a4522d9563fe5 not found: ID does not exist"
Oct 29 23:24:43.663895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.663827    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"]
Oct 29 23:24:43.669599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.669576    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-pjtcr"]
Oct 29 23:24:43.882934 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:43.882903    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:44.574284 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:44.574255    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"
Oct 29 23:24:44.780106 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:44.780076    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a967af1-6e08-4983-9493-0292d3ea8aaa" path="/var/lib/kubelet/pods/1a967af1-6e08-4983-9493-0292d3ea8aaa/volumes"
Oct 29 23:24:45.104405 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:45.104373    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:46.901567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:46.901542    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:47.174102 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.174078    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 23:24:47.213352 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.213320    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-credential-socket\") pod \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") "
Oct 29 23:24:47.213514 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.213391    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istiod-ca-cert\") pod \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") "
Oct 29 23:24:47.213514 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.213493    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-envoy\") pod \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") "
Oct 29 23:24:47.213643 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.213569    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-podinfo\") pod \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") "
Oct 29 23:24:47.213643 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.213635    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-credential-socket" pod "ae1fd856-9cb0-4d65-bc25-49aefe12e84a" (UID: "ae1fd856-9cb0-4d65-bc25-49aefe12e84a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:47.213763 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.213736    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-workload-certs\") pod \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") "
Oct 29 23:24:47.213846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.213826    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istiod-ca-cert" pod "ae1fd856-9cb0-4d65-bc25-49aefe12e84a" (UID: "ae1fd856-9cb0-4d65-bc25-49aefe12e84a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:24:47.213912 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.213835    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-kube-api-access-wqvbd\" (UniqueName: \"kubernetes.io/projected/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-kube-api-access-wqvbd\") pod \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") "
Oct 29 23:24:47.213978 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.213917    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-token\" (UniqueName: \"kubernetes.io/projected/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-token\") pod \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") "
Oct 29 23:24:47.214039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.213992    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-workload-socket\") pod \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") "
Oct 29 23:24:47.214136 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.214054    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-data\") pod \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\" (UID: \"ae1fd856-9cb0-4d65-bc25-49aefe12e84a\") "
Oct 29 23:24:47.214396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.214364    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-workload-socket" pod "ae1fd856-9cb0-4d65-bc25-49aefe12e84a" (UID: "ae1fd856-9cb0-4d65-bc25-49aefe12e84a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:47.214518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.214485    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-workload-certs" pod "ae1fd856-9cb0-4d65-bc25-49aefe12e84a" (UID: "ae1fd856-9cb0-4d65-bc25-49aefe12e84a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:47.214648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.214617    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-data" pod "ae1fd856-9cb0-4d65-bc25-49aefe12e84a" (UID: "ae1fd856-9cb0-4d65-bc25-49aefe12e84a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:47.215032 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.215005    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:47.215134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.215036    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:47.215134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.215051    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:47.215134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.215066    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:47.215134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.215084    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:47.216240 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.216199    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-podinfo" pod "ae1fd856-9cb0-4d65-bc25-49aefe12e84a" (UID: "ae1fd856-9cb0-4d65-bc25-49aefe12e84a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:24:47.216412 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.216393    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-envoy" pod "ae1fd856-9cb0-4d65-bc25-49aefe12e84a" (UID: "ae1fd856-9cb0-4d65-bc25-49aefe12e84a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:24:47.216657 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.216628    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-token" pod "ae1fd856-9cb0-4d65-bc25-49aefe12e84a" (UID: "ae1fd856-9cb0-4d65-bc25-49aefe12e84a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:47.217018 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.216991    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-kube-api-access-wqvbd" pod "ae1fd856-9cb0-4d65-bc25-49aefe12e84a" (UID: "ae1fd856-9cb0-4d65-bc25-49aefe12e84a"). InnerVolumeSpecName "kube-api-access-wqvbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:47.316355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.316325    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:47.316355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.316352    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:47.316355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.316362    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:47.316601 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.316370    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqvbd\" (UniqueName: \"kubernetes.io/projected/ae1fd856-9cb0-4d65-bc25-49aefe12e84a-kube-api-access-wqvbd\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:24:47.371120 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:24:47.371083    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:24:47.649507 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.649466    5428 generic.go:356] "Generic (PLEG): container finished" podID="ae1fd856-9cb0-4d65-bc25-49aefe12e84a" containerID="c783e485126991a6cb5dc6acbebe6b1e208c9377dd1b110e7e90a1cf8862e358" exitCode=0
Oct 29 23:24:47.649690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.649531    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"
Oct 29 23:24:47.649690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.649543    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd" event={"ID":"ae1fd856-9cb0-4d65-bc25-49aefe12e84a","Type":"ContainerDied","Data":"c783e485126991a6cb5dc6acbebe6b1e208c9377dd1b110e7e90a1cf8862e358"}
Oct 29 23:24:47.649690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.649616    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd" event={"ID":"ae1fd856-9cb0-4d65-bc25-49aefe12e84a","Type":"ContainerDied","Data":"157639e6794c0cf21cd5a66791e767d72d0e770b1eccc63b244f75acb687dd73"}
Oct 29 23:24:47.649690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.649639    5428 scope.go:122] "RemoveContainer" containerID="c783e485126991a6cb5dc6acbebe6b1e208c9377dd1b110e7e90a1cf8862e358"
Oct 29 23:24:47.664617 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.664600    5428 scope.go:122] "RemoveContainer" containerID="4dfd90cc1c3f82298cffd572d81380e54c81d3b035b41168b1a0c350092c9659"
Oct 29 23:24:47.676073 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.676056    5428 scope.go:122] "RemoveContainer" containerID="c783e485126991a6cb5dc6acbebe6b1e208c9377dd1b110e7e90a1cf8862e358"
Oct 29 23:24:47.676383 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:24:47.676351    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c783e485126991a6cb5dc6acbebe6b1e208c9377dd1b110e7e90a1cf8862e358\": container with ID starting with c783e485126991a6cb5dc6acbebe6b1e208c9377dd1b110e7e90a1cf8862e358 not found: ID does not exist" containerID="c783e485126991a6cb5dc6acbebe6b1e208c9377dd1b110e7e90a1cf8862e358"
Oct 29 23:24:47.676460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.676393    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c783e485126991a6cb5dc6acbebe6b1e208c9377dd1b110e7e90a1cf8862e358"} err="failed to get container status \"c783e485126991a6cb5dc6acbebe6b1e208c9377dd1b110e7e90a1cf8862e358\": rpc error: code = NotFound desc = could not find container \"c783e485126991a6cb5dc6acbebe6b1e208c9377dd1b110e7e90a1cf8862e358\": container with ID starting with c783e485126991a6cb5dc6acbebe6b1e208c9377dd1b110e7e90a1cf8862e358 not found: ID does not exist"
Oct 29 23:24:47.676460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.676410    5428 scope.go:122] "RemoveContainer" containerID="4dfd90cc1c3f82298cffd572d81380e54c81d3b035b41168b1a0c350092c9659"
Oct 29 23:24:47.676655 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:24:47.676634    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dfd90cc1c3f82298cffd572d81380e54c81d3b035b41168b1a0c350092c9659\": container with ID starting with 4dfd90cc1c3f82298cffd572d81380e54c81d3b035b41168b1a0c350092c9659 not found: ID does not exist" containerID="4dfd90cc1c3f82298cffd572d81380e54c81d3b035b41168b1a0c350092c9659"
Oct 29 23:24:47.676717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.676660    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dfd90cc1c3f82298cffd572d81380e54c81d3b035b41168b1a0c350092c9659"} err="failed to get container status \"4dfd90cc1c3f82298cffd572d81380e54c81d3b035b41168b1a0c350092c9659\": rpc error: code = NotFound desc = could not find container \"4dfd90cc1c3f82298cffd572d81380e54c81d3b035b41168b1a0c350092c9659\": container with ID starting with 4dfd90cc1c3f82298cffd572d81380e54c81d3b035b41168b1a0c350092c9659 not found: ID does not exist"
Oct 29 23:24:47.703308 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.703280    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"]
Oct 29 23:24:47.715481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:47.715457    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed8218-openshift-default-676f9757-nbwdd"]
Oct 29 23:24:48.724672 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:48.724638    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:48.790642 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:48.790608    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae1fd856-9cb0-4d65-bc25-49aefe12e84a" path="/var/lib/kubelet/pods/ae1fd856-9cb0-4d65-bc25-49aefe12e84a/volumes"
Oct 29 23:24:49.572578 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:49.572549    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt"
Oct 29 23:24:50.505565 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:50.505531    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:52.197083 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:52.197053    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:53.180101 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:53.180062    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:54.300403 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:54.300372    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:56.215016 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:56.214975    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:24:58.200967 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:24:58.200932    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:00.061743 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:00.061711    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:01.907015 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:01.906981    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:03.682185 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:03.682140    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:03.907211 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:03.907170    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf"]
Oct 29 23:25:03.938712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:03.938625    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8"]
Oct 29 23:25:04.046700 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:04.046665    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:04.168727 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:04.168699    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:04.650571 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:04.650540    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:04.878507 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:04.878476    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:05.576911 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:05.576877    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:07.407318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:07.407283    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:08.908306 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:08.908247    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf" podUID="fd5901f6-73be-4fd8-b7c1-d9237b6e934a" containerName="payload-processing" containerID="cri-o://4c95d3c055065e02d005397f3c0a6aa8b7a7cfadaa26b2790e71de4da2ee5fba" gracePeriod=25
Oct 29 23:25:08.939125 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:08.939088    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8" podUID="07b3da27-929f-471c-a627-1ac5e4af844f" containerName="payload-pre-processing" containerID="cri-o://e7d96d69884541d6297bb0ec6dda9bd64b81fac73b9fa822e6da7377ce8cc9e7" gracePeriod=25
Oct 29 23:25:09.000677 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.000643    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:09.283096 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.283059    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf"
Oct 29 23:25:09.360305 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.360271    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8"
Oct 29 23:25:09.380777 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.380737    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd5901f6-73be-4fd8-b7c1-d9237b6e934a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5901f6-73be-4fd8-b7c1-d9237b6e934a-plugins-config-volume\") pod \"fd5901f6-73be-4fd8-b7c1-d9237b6e934a\" (UID: \"fd5901f6-73be-4fd8-b7c1-d9237b6e934a\") "
Oct 29 23:25:09.380983 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.380831    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5901f6-73be-4fd8-b7c1-d9237b6e934a-kube-api-access-9frph\" (UniqueName: \"kubernetes.io/projected/fd5901f6-73be-4fd8-b7c1-d9237b6e934a-kube-api-access-9frph\") pod \"fd5901f6-73be-4fd8-b7c1-d9237b6e934a\" (UID: \"fd5901f6-73be-4fd8-b7c1-d9237b6e934a\") "
Oct 29 23:25:09.381126 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.381106    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b3da27-929f-471c-a627-1ac5e4af844f-kube-api-access-rlpwm\" (UniqueName: \"kubernetes.io/projected/07b3da27-929f-471c-a627-1ac5e4af844f-kube-api-access-rlpwm\") pod \"07b3da27-929f-471c-a627-1ac5e4af844f\" (UID: \"07b3da27-929f-471c-a627-1ac5e4af844f\") "
Oct 29 23:25:09.381211 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.381153    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b3da27-929f-471c-a627-1ac5e4af844f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b3da27-929f-471c-a627-1ac5e4af844f-plugins-config-volume\") pod \"07b3da27-929f-471c-a627-1ac5e4af844f\" (UID: \"07b3da27-929f-471c-a627-1ac5e4af844f\") "
Oct 29 23:25:09.381211 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.381184    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5901f6-73be-4fd8-b7c1-d9237b6e934a-plugins-config-volume" pod "fd5901f6-73be-4fd8-b7c1-d9237b6e934a" (UID: "fd5901f6-73be-4fd8-b7c1-d9237b6e934a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:25:09.381587 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.381557    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b3da27-929f-471c-a627-1ac5e4af844f-plugins-config-volume" pod "07b3da27-929f-471c-a627-1ac5e4af844f" (UID: "07b3da27-929f-471c-a627-1ac5e4af844f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:25:09.381730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.381711    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b3da27-929f-471c-a627-1ac5e4af844f-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:25:09.381820 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.381736    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5901f6-73be-4fd8-b7c1-d9237b6e934a-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:25:09.384577 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.384547    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b3da27-929f-471c-a627-1ac5e4af844f-kube-api-access-rlpwm" pod "07b3da27-929f-471c-a627-1ac5e4af844f" (UID: "07b3da27-929f-471c-a627-1ac5e4af844f"). InnerVolumeSpecName "kube-api-access-rlpwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:25:09.385951 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.385924    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5901f6-73be-4fd8-b7c1-d9237b6e934a-kube-api-access-9frph" pod "fd5901f6-73be-4fd8-b7c1-d9237b6e934a" (UID: "fd5901f6-73be-4fd8-b7c1-d9237b6e934a"). InnerVolumeSpecName "kube-api-access-9frph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:25:09.489161 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.489057    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlpwm\" (UniqueName: \"kubernetes.io/projected/07b3da27-929f-471c-a627-1ac5e4af844f-kube-api-access-rlpwm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:25:09.489161 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.489099    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9frph\" (UniqueName: \"kubernetes.io/projected/fd5901f6-73be-4fd8-b7c1-d9237b6e934a-kube-api-access-9frph\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:25:09.695990 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.695959    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:09.842002 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.841925    5428 generic.go:356] "Generic (PLEG): container finished" podID="fd5901f6-73be-4fd8-b7c1-d9237b6e934a" containerID="4c95d3c055065e02d005397f3c0a6aa8b7a7cfadaa26b2790e71de4da2ee5fba" exitCode=0
Oct 29 23:25:09.842177 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.842010    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf"
Oct 29 23:25:09.842177 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.842039    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf" event={"ID":"fd5901f6-73be-4fd8-b7c1-d9237b6e934a","Type":"ContainerDied","Data":"4c95d3c055065e02d005397f3c0a6aa8b7a7cfadaa26b2790e71de4da2ee5fba"}
Oct 29 23:25:09.842177 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.842094    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf" event={"ID":"fd5901f6-73be-4fd8-b7c1-d9237b6e934a","Type":"ContainerDied","Data":"01d81a2be912941f87523254b84d60ed56f2782d31f97d7bbc5edd49ee7a8554"}
Oct 29 23:25:09.842177 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.842125    5428 scope.go:122] "RemoveContainer" containerID="4c95d3c055065e02d005397f3c0a6aa8b7a7cfadaa26b2790e71de4da2ee5fba"
Oct 29 23:25:09.843667 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.843637    5428 generic.go:356] "Generic (PLEG): container finished" podID="07b3da27-929f-471c-a627-1ac5e4af844f" containerID="e7d96d69884541d6297bb0ec6dda9bd64b81fac73b9fa822e6da7377ce8cc9e7" exitCode=0
Oct 29 23:25:09.843782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.843689    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8"
Oct 29 23:25:09.843782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.843722    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8" event={"ID":"07b3da27-929f-471c-a627-1ac5e4af844f","Type":"ContainerDied","Data":"e7d96d69884541d6297bb0ec6dda9bd64b81fac73b9fa822e6da7377ce8cc9e7"}
Oct 29 23:25:09.843782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.843765    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8" event={"ID":"07b3da27-929f-471c-a627-1ac5e4af844f","Type":"ContainerDied","Data":"c3e2b93a332f019451fecb19db9ee1089b2d2c5288a25cb51d844c69b4e9f9ee"}
Oct 29 23:25:09.855418 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.855404    5428 scope.go:122] "RemoveContainer" containerID="4c95d3c055065e02d005397f3c0a6aa8b7a7cfadaa26b2790e71de4da2ee5fba"
Oct 29 23:25:09.855688 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:25:09.855668    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c95d3c055065e02d005397f3c0a6aa8b7a7cfadaa26b2790e71de4da2ee5fba\": container with ID starting with 4c95d3c055065e02d005397f3c0a6aa8b7a7cfadaa26b2790e71de4da2ee5fba not found: ID does not exist" containerID="4c95d3c055065e02d005397f3c0a6aa8b7a7cfadaa26b2790e71de4da2ee5fba"
Oct 29 23:25:09.855752 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.855694    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c95d3c055065e02d005397f3c0a6aa8b7a7cfadaa26b2790e71de4da2ee5fba"} err="failed to get container status \"4c95d3c055065e02d005397f3c0a6aa8b7a7cfadaa26b2790e71de4da2ee5fba\": rpc error: code = NotFound desc = could not find container \"4c95d3c055065e02d005397f3c0a6aa8b7a7cfadaa26b2790e71de4da2ee5fba\": container with ID starting with 4c95d3c055065e02d005397f3c0a6aa8b7a7cfadaa26b2790e71de4da2ee5fba not found: ID does not exist"
Oct 29 23:25:09.855752 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.855710    5428 scope.go:122] "RemoveContainer" containerID="e7d96d69884541d6297bb0ec6dda9bd64b81fac73b9fa822e6da7377ce8cc9e7"
Oct 29 23:25:09.865733 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.865719    5428 scope.go:122] "RemoveContainer" containerID="e7d96d69884541d6297bb0ec6dda9bd64b81fac73b9fa822e6da7377ce8cc9e7"
Oct 29 23:25:09.865974 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:25:09.865952    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d96d69884541d6297bb0ec6dda9bd64b81fac73b9fa822e6da7377ce8cc9e7\": container with ID starting with e7d96d69884541d6297bb0ec6dda9bd64b81fac73b9fa822e6da7377ce8cc9e7 not found: ID does not exist" containerID="e7d96d69884541d6297bb0ec6dda9bd64b81fac73b9fa822e6da7377ce8cc9e7"
Oct 29 23:25:09.866027 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.865981    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d96d69884541d6297bb0ec6dda9bd64b81fac73b9fa822e6da7377ce8cc9e7"} err="failed to get container status \"e7d96d69884541d6297bb0ec6dda9bd64b81fac73b9fa822e6da7377ce8cc9e7\": rpc error: code = NotFound desc = could not find container \"e7d96d69884541d6297bb0ec6dda9bd64b81fac73b9fa822e6da7377ce8cc9e7\": container with ID starting with e7d96d69884541d6297bb0ec6dda9bd64b81fac73b9fa822e6da7377ce8cc9e7 not found: ID does not exist"
Oct 29 23:25:09.924313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.924284    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf"]
Oct 29 23:25:09.938115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.938090    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c1e164-6f7996656-pw8tf"]
Oct 29 23:25:09.993858 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.993831    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8"]
Oct 29 23:25:09.998121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:09.998098    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c1e164-6454858675-n44t8"]
Oct 29 23:25:10.030039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:10.030008    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:10.139457 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:10.139421    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:10.536682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:10.536601    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:10.779218 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:10.779184    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07b3da27-929f-471c-a627-1ac5e4af844f" path="/var/lib/kubelet/pods/07b3da27-929f-471c-a627-1ac5e4af844f/volumes"
Oct 29 23:25:10.779680 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:10.779665    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd5901f6-73be-4fd8-b7c1-d9237b6e934a" path="/var/lib/kubelet/pods/fd5901f6-73be-4fd8-b7c1-d9237b6e934a/volumes"
Oct 29 23:25:12.209734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:12.209696    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:13.944955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.944921    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"]
Oct 29 23:25:13.945738 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.945722    5428 cpu_manager.go:414] "containerMap: removing container" podUID="07b3da27-929f-471c-a627-1ac5e4af844f" containerName="payload-pre-processing"
Oct 29 23:25:13.945785 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.945739    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b3da27-929f-471c-a627-1ac5e4af844f" containerName="payload-pre-processing"
Oct 29 23:25:13.945785 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.945765    5428 cpu_manager.go:414] "containerMap: removing container" podUID="fd5901f6-73be-4fd8-b7c1-d9237b6e934a" containerName="payload-processing"
Oct 29 23:25:13.945785 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.945770    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd5901f6-73be-4fd8-b7c1-d9237b6e934a" containerName="payload-processing"
Oct 29 23:25:13.945785 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.945782    5428 cpu_manager.go:414] "containerMap: removing container" podUID="ae1fd856-9cb0-4d65-bc25-49aefe12e84a" containerName="istio-proxy"
Oct 29 23:25:13.945941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.945788    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1fd856-9cb0-4d65-bc25-49aefe12e84a" containerName="istio-proxy"
Oct 29 23:25:13.945941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.945816    5428 cpu_manager.go:414] "containerMap: removing container" podUID="1a967af1-6e08-4983-9493-0292d3ea8aaa" containerName="istio-proxy"
Oct 29 23:25:13.945941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.945822    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a967af1-6e08-4983-9493-0292d3ea8aaa" containerName="istio-proxy"
Oct 29 23:25:13.945941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.945833    5428 cpu_manager.go:414] "containerMap: removing container" podUID="ae1fd856-9cb0-4d65-bc25-49aefe12e84a" containerName="istio-proxy"
Oct 29 23:25:13.945941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.945838    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1fd856-9cb0-4d65-bc25-49aefe12e84a" containerName="istio-proxy"
Oct 29 23:25:13.946128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.945959    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b3da27-929f-471c-a627-1ac5e4af844f" containerName="payload-pre-processing"
Oct 29 23:25:13.946128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.945969    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd5901f6-73be-4fd8-b7c1-d9237b6e934a" containerName="payload-processing"
Oct 29 23:25:13.946128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.945976    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1fd856-9cb0-4d65-bc25-49aefe12e84a" containerName="istio-proxy"
Oct 29 23:25:13.946128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.945983    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a967af1-6e08-4983-9493-0292d3ea8aaa" containerName="istio-proxy"
Oct 29 23:25:13.946128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.945989    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1fd856-9cb0-4d65-bc25-49aefe12e84a" containerName="istio-proxy"
Oct 29 23:25:13.949701 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.949681    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"
Oct 29 23:25:13.967564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.967542    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-fb367d-serving-cert\""
Oct 29 23:25:13.984212 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:13.984184    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"]
Oct 29 23:25:14.044651 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:14.044583    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:14.058148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:14.058121    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghbpr\" (UniqueName: \"kubernetes.io/projected/b546f7f7-0c2c-4da9-a00d-41b52bb85815-kube-api-access-ghbpr\") pod \"maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72\" (UID: \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"
Oct 29 23:25:14.058310 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:14.058238    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b546f7f7-0c2c-4da9-a00d-41b52bb85815-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72\" (UID: \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"
Oct 29 23:25:14.058352 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:14.058320    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b546f7f7-0c2c-4da9-a00d-41b52bb85815-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72\" (UID: \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"
Oct 29 23:25:14.058448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:14.058429    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b546f7f7-0c2c-4da9-a00d-41b52bb85815-metrics-certs\") pod \"maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72\" (UID: \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"
Oct 29 23:25:14.159456 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:14.159420    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghbpr\" (UniqueName: \"kubernetes.io/projected/b546f7f7-0c2c-4da9-a00d-41b52bb85815-kube-api-access-ghbpr\") pod \"maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72\" (UID: \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"
Oct 29 23:25:14.159648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:14.159527    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b546f7f7-0c2c-4da9-a00d-41b52bb85815-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72\" (UID: \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"
Oct 29 23:25:14.159648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:14.159577    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b546f7f7-0c2c-4da9-a00d-41b52bb85815-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72\" (UID: \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"
Oct 29 23:25:14.159648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:14.159635    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b546f7f7-0c2c-4da9-a00d-41b52bb85815-metrics-certs\") pod \"maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72\" (UID: \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"
Oct 29 23:25:14.160221 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:14.160183    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b546f7f7-0c2c-4da9-a00d-41b52bb85815-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72\" (UID: \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"
Oct 29 23:25:14.160898 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:14.160872    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:14.161974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:14.161956    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b546f7f7-0c2c-4da9-a00d-41b52bb85815-metrics-certs\") pod \"maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72\" (UID: \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"
Oct 29 23:25:14.162106 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:14.162088    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b546f7f7-0c2c-4da9-a00d-41b52bb85815-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72\" (UID: \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"
Oct 29 23:25:14.185394 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:14.185368    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghbpr\" (UniqueName: \"kubernetes.io/projected/b546f7f7-0c2c-4da9-a00d-41b52bb85815-kube-api-access-ghbpr\") pod \"maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72\" (UID: \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"
Oct 29 23:25:14.264453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:14.264369    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"
Oct 29 23:25:14.282653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:14.282621    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:14.578721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:14.578692    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"]
Oct 29 23:25:14.579338 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:25:14.579315    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb546f7f7_0c2c_4da9_a00d_41b52bb85815.slice/crio-56e272e861491919b4370fc94e9e5e2b3970ca7f91643a68d37b8dedc4f2ef3e WatchSource:0}: Error finding container 56e272e861491919b4370fc94e9e5e2b3970ca7f91643a68d37b8dedc4f2ef3e: Status 404 returned error can't find the container with id 56e272e861491919b4370fc94e9e5e2b3970ca7f91643a68d37b8dedc4f2ef3e
Oct 29 23:25:14.889616 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:14.889586    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72" event={"ID":"b546f7f7-0c2c-4da9-a00d-41b52bb85815","Type":"ContainerStarted","Data":"56e272e861491919b4370fc94e9e5e2b3970ca7f91643a68d37b8dedc4f2ef3e"}
Oct 29 23:25:15.868336 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:15.868298    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:15.901428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:15.901394    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72" event={"ID":"b546f7f7-0c2c-4da9-a00d-41b52bb85815","Type":"ContainerStarted","Data":"20b4af7b06c5d36f49638f68e7e66f7c47531c18b5baec3a56ef194c8d6090cb"}
Oct 29 23:25:15.901640 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:15.901498    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"
Oct 29 23:25:15.949944 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:15.949866    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72" podStartSLOduration=2.653751019 podStartE2EDuration="2.949842993s" totalImagesPullingTime="296.091974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:15.944277591 +0000 UTC m=+2519.287226337" watchObservedRunningTime="2026-10-29 23:25:15.949842993 +0000 UTC m=+2519.292791738"
Oct 29 23:25:17.339383 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:17.339349    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:17.369057 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:25:17.369024    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:25:17.464463 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:17.464429    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:17.861419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:17.861376    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:18.215742 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:18.215712    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:19.653601 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:19.653567    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:20.136759 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:20.136729    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:20.635657 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:20.635622    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:21.032921 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:21.032827    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:21.917230 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:21.917200    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"
Oct 29 23:25:23.097965 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:23.097928    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:24.582614 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:24.582586    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:24.773154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:24.773121    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:25.254667 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:25.254633    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:26.244229 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.244195    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:26.383974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.383939    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:26.418844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.418792    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg"]
Oct 29 23:25:26.425296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.425280    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg"
Oct 29 23:25:26.443451 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.443426    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg"]
Oct 29 23:25:26.448921 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.448894    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-45036b\""
Oct 29 23:25:26.449070 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.448908    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-45036b-dockercfg-z8gjj\""
Oct 29 23:25:26.563228 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.563144    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh"]
Oct 29 23:25:26.568607 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.568559    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9205c9d1-7bc4-45ea-908f-776fb46a4bbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg\" (UID: \"9205c9d1-7bc4-45ea-908f-776fb46a4bbe\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg"
Oct 29 23:25:26.568975 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.568948    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s7ws\" (UniqueName: \"kubernetes.io/projected/9205c9d1-7bc4-45ea-908f-776fb46a4bbe-kube-api-access-5s7ws\") pod \"payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg\" (UID: \"9205c9d1-7bc4-45ea-908f-776fb46a4bbe\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg"
Oct 29 23:25:26.579376 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.579353    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh"
Oct 29 23:25:26.585288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.584755    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh"]
Oct 29 23:25:26.670441 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.670396    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9205c9d1-7bc4-45ea-908f-776fb46a4bbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg\" (UID: \"9205c9d1-7bc4-45ea-908f-776fb46a4bbe\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg"
Oct 29 23:25:26.670652 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.670475    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s7ws\" (UniqueName: \"kubernetes.io/projected/9205c9d1-7bc4-45ea-908f-776fb46a4bbe-kube-api-access-5s7ws\") pod \"payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg\" (UID: \"9205c9d1-7bc4-45ea-908f-776fb46a4bbe\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg"
Oct 29 23:25:26.672239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.672207    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9205c9d1-7bc4-45ea-908f-776fb46a4bbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg\" (UID: \"9205c9d1-7bc4-45ea-908f-776fb46a4bbe\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg"
Oct 29 23:25:26.695323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.695298    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s7ws\" (UniqueName: \"kubernetes.io/projected/9205c9d1-7bc4-45ea-908f-776fb46a4bbe-kube-api-access-5s7ws\") pod \"payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg\" (UID: \"9205c9d1-7bc4-45ea-908f-776fb46a4bbe\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg"
Oct 29 23:25:26.741128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.741100    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg"
Oct 29 23:25:26.772595 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.772557    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df4wl\" (UniqueName: \"kubernetes.io/projected/b1ac24ef-1667-4022-be72-55ffbd459242-kube-api-access-df4wl\") pod \"payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh\" (UID: \"b1ac24ef-1667-4022-be72-55ffbd459242\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh"
Oct 29 23:25:26.773022 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.772998    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ac24ef-1667-4022-be72-55ffbd459242-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh\" (UID: \"b1ac24ef-1667-4022-be72-55ffbd459242\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh"
Oct 29 23:25:26.874638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.874193    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ac24ef-1667-4022-be72-55ffbd459242-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh\" (UID: \"b1ac24ef-1667-4022-be72-55ffbd459242\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh"
Oct 29 23:25:26.874638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.874284    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df4wl\" (UniqueName: \"kubernetes.io/projected/b1ac24ef-1667-4022-be72-55ffbd459242-kube-api-access-df4wl\") pod \"payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh\" (UID: \"b1ac24ef-1667-4022-be72-55ffbd459242\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh"
Oct 29 23:25:26.875304 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.875277    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ac24ef-1667-4022-be72-55ffbd459242-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh\" (UID: \"b1ac24ef-1667-4022-be72-55ffbd459242\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh"
Oct 29 23:25:26.897613 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.897587    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df4wl\" (UniqueName: \"kubernetes.io/projected/b1ac24ef-1667-4022-be72-55ffbd459242-kube-api-access-df4wl\") pod \"payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh\" (UID: \"b1ac24ef-1667-4022-be72-55ffbd459242\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh"
Oct 29 23:25:26.901567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:26.901540    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh"
Oct 29 23:25:27.052436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:27.052409    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg"]
Oct 29 23:25:27.053906 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:25:27.053876    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9205c9d1_7bc4_45ea_908f_776fb46a4bbe.slice/crio-87c39fe7bfc69ac43520d0e2bc3fb7f83533b254f76f08024bb159b96e901472 WatchSource:0}: Error finding container 87c39fe7bfc69ac43520d0e2bc3fb7f83533b254f76f08024bb159b96e901472: Status 404 returned error can't find the container with id 87c39fe7bfc69ac43520d0e2bc3fb7f83533b254f76f08024bb159b96e901472
Oct 29 23:25:27.217244 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:27.217221    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh"]
Oct 29 23:25:27.217918 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:25:27.217892    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ac24ef_1667_4022_be72_55ffbd459242.slice/crio-40e27bddb35bc789d75121389c01e330096e6fab1cc6ff55b06bb5b548379099 WatchSource:0}: Error finding container 40e27bddb35bc789d75121389c01e330096e6fab1cc6ff55b06bb5b548379099: Status 404 returned error can't find the container with id 40e27bddb35bc789d75121389c01e330096e6fab1cc6ff55b06bb5b548379099
Oct 29 23:25:27.495651 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:27.495574    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:28.008059 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:28.008025    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg" event={"ID":"9205c9d1-7bc4-45ea-908f-776fb46a4bbe","Type":"ContainerStarted","Data":"97d23b71bd3f362e442a65041c8da631d9d398aa0dce723afbc28235646bbe65"}
Oct 29 23:25:28.008269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:28.008084    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg" event={"ID":"9205c9d1-7bc4-45ea-908f-776fb46a4bbe","Type":"ContainerStarted","Data":"87c39fe7bfc69ac43520d0e2bc3fb7f83533b254f76f08024bb159b96e901472"}
Oct 29 23:25:28.008269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:28.008116    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg"
Oct 29 23:25:28.009945 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:28.009919    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh" event={"ID":"b1ac24ef-1667-4022-be72-55ffbd459242","Type":"ContainerStarted","Data":"f4213bd68d0cd6abecaee77fe95a73a2c74008d8aad4455e01614be0603dde9f"}
Oct 29 23:25:28.010075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:28.009958    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh" event={"ID":"b1ac24ef-1667-4022-be72-55ffbd459242","Type":"ContainerStarted","Data":"40e27bddb35bc789d75121389c01e330096e6fab1cc6ff55b06bb5b548379099"}
Oct 29 23:25:28.010075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:28.010023    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh"
Oct 29 23:25:28.059488 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:28.059441    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg" podStartSLOduration=2.059426955 podStartE2EDuration="2.059426955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:28.05592602 +0000 UTC m=+2531.398874765" watchObservedRunningTime="2026-10-29 23:25:28.059426955 +0000 UTC m=+2531.402375725"
Oct 29 23:25:28.100958 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:28.100903    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh" podStartSLOduration=2.100885315 podStartE2EDuration="2.100885315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:28.095342874 +0000 UTC m=+2531.438291629" watchObservedRunningTime="2026-10-29 23:25:28.100885315 +0000 UTC m=+2531.443834067"
Oct 29 23:25:29.165859 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.165828    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"]
Oct 29 23:25:29.171107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.171090    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:25:29.198008 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.197970    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-7d4110-serving-cert\""
Oct 29 23:25:29.202992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.202964    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"]
Oct 29 23:25:29.205906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.205882    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e14ce23-ae46-4db5-9275-ceaf9987ba61-metrics-certs\") pod \"maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj\" (UID: \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:25:29.206037 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.206018    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfths\" (UniqueName: \"kubernetes.io/projected/9e14ce23-ae46-4db5-9275-ceaf9987ba61-kube-api-access-bfths\") pod \"maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj\" (UID: \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:25:29.206107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.206067    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e14ce23-ae46-4db5-9275-ceaf9987ba61-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj\" (UID: \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:25:29.206303 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.206280    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e14ce23-ae46-4db5-9275-ceaf9987ba61-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj\" (UID: \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:25:29.266209 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.266180    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:29.308081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.308048    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfths\" (UniqueName: \"kubernetes.io/projected/9e14ce23-ae46-4db5-9275-ceaf9987ba61-kube-api-access-bfths\") pod \"maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj\" (UID: \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:25:29.308258 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.308097    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e14ce23-ae46-4db5-9275-ceaf9987ba61-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj\" (UID: \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:25:29.308315 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.308279    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e14ce23-ae46-4db5-9275-ceaf9987ba61-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj\" (UID: \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:25:29.308436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.308417    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e14ce23-ae46-4db5-9275-ceaf9987ba61-metrics-certs\") pod \"maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj\" (UID: \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:25:29.308577 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:25:29.308425    5428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-serving-cert: secret "maas-api-e2e-extmod-b-7d4110-serving-cert" not found
Oct 29 23:25:29.308676 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:25:29.308664    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e14ce23-ae46-4db5-9275-ceaf9987ba61-maas-api-tls podName:9e14ce23-ae46-4db5-9275-ceaf9987ba61 nodeName:}" failed. No retries permitted until 2026-10-29 23:25:29.808648144 +0000 UTC m=+2533.151596871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e14ce23-ae46-4db5-9275-ceaf9987ba61-maas-api-tls") pod "maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj" (UID: "9e14ce23-ae46-4db5-9275-ceaf9987ba61") : secret "maas-api-e2e-extmod-b-7d4110-serving-cert" not found
Oct 29 23:25:29.308793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.308773    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e14ce23-ae46-4db5-9275-ceaf9987ba61-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj\" (UID: \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:25:29.310808 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.310785    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e14ce23-ae46-4db5-9275-ceaf9987ba61-metrics-certs\") pod \"maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj\" (UID: \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:25:29.331712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.331689    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfths\" (UniqueName: \"kubernetes.io/projected/9e14ce23-ae46-4db5-9275-ceaf9987ba61-kube-api-access-bfths\") pod \"maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj\" (UID: \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:25:29.389602 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.389568    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:29.503738 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.503658    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:29.815382 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.815296    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e14ce23-ae46-4db5-9275-ceaf9987ba61-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj\" (UID: \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:25:29.817868 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:29.817841    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e14ce23-ae46-4db5-9275-ceaf9987ba61-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj\" (UID: \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:25:30.088476 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:30.088375    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:25:30.224402 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:30.224370    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:30.439591 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:30.439555    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"]
Oct 29 23:25:31.036900 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:31.036781    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj" event={"ID":"9e14ce23-ae46-4db5-9275-ceaf9987ba61","Type":"ContainerStarted","Data":"518ace8c14d11f4531e1abb2a19454bea5aa755fcc8e63376ee89c8381710104"}
Oct 29 23:25:31.036900 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:31.036869    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj" event={"ID":"9e14ce23-ae46-4db5-9275-ceaf9987ba61","Type":"ContainerStarted","Data":"626e4450dc091c2896768a490aaf9c202820caadb12c78f1d026ef95d08edd4f"}
Oct 29 23:25:31.036900 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:31.036900    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:25:31.082403 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:31.082354    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj" podStartSLOduration=1.779048216 podStartE2EDuration="2.082340148s" totalImagesPullingTime="303.291932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:31.07854233 +0000 UTC m=+2534.421491076" watchObservedRunningTime="2026-10-29 23:25:31.082340148 +0000 UTC m=+2534.425288933"
Oct 29 23:25:31.194569 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:31.194534    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:33.042666 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:33.042633    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:33.987765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:33.987728    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:34.101477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:34.101448    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:34.651259 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:34.651227    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:35.354722 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:35.354684    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:37.052059 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.052029    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:25:37.145243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.145209    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"]
Oct 29 23:25:37.150601 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.150577    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"
Oct 29 23:25:37.168430 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.168399    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-d5e482-serving-cert\""
Oct 29 23:25:37.182840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.179968    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"]
Oct 29 23:25:37.240840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.240791    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4ba52cf-6aac-4495-810c-8ba495c72d70-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t\" (UID: \"e4ba52cf-6aac-4495-810c-8ba495c72d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"
Oct 29 23:25:37.241018 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.240941    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8kzm\" (UniqueName: \"kubernetes.io/projected/e4ba52cf-6aac-4495-810c-8ba495c72d70-kube-api-access-h8kzm\") pod \"maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t\" (UID: \"e4ba52cf-6aac-4495-810c-8ba495c72d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"
Oct 29 23:25:37.241086 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.241037    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ba52cf-6aac-4495-810c-8ba495c72d70-metrics-certs\") pod \"maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t\" (UID: \"e4ba52cf-6aac-4495-810c-8ba495c72d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"
Oct 29 23:25:37.241194 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.241172    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ba52cf-6aac-4495-810c-8ba495c72d70-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t\" (UID: \"e4ba52cf-6aac-4495-810c-8ba495c72d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"
Oct 29 23:25:37.248250 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.248225    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:37.343290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.343211    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8kzm\" (UniqueName: \"kubernetes.io/projected/e4ba52cf-6aac-4495-810c-8ba495c72d70-kube-api-access-h8kzm\") pod \"maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t\" (UID: \"e4ba52cf-6aac-4495-810c-8ba495c72d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"
Oct 29 23:25:37.343452 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.343290    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ba52cf-6aac-4495-810c-8ba495c72d70-metrics-certs\") pod \"maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t\" (UID: \"e4ba52cf-6aac-4495-810c-8ba495c72d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"
Oct 29 23:25:37.343452 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.343362    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ba52cf-6aac-4495-810c-8ba495c72d70-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t\" (UID: \"e4ba52cf-6aac-4495-810c-8ba495c72d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"
Oct 29 23:25:37.343596 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.343578    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4ba52cf-6aac-4495-810c-8ba495c72d70-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t\" (UID: \"e4ba52cf-6aac-4495-810c-8ba495c72d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"
Oct 29 23:25:37.344091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.344067    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ba52cf-6aac-4495-810c-8ba495c72d70-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t\" (UID: \"e4ba52cf-6aac-4495-810c-8ba495c72d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"
Oct 29 23:25:37.346504 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.346396    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4ba52cf-6aac-4495-810c-8ba495c72d70-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t\" (UID: \"e4ba52cf-6aac-4495-810c-8ba495c72d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"
Oct 29 23:25:37.346653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.346624    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ba52cf-6aac-4495-810c-8ba495c72d70-metrics-certs\") pod \"maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t\" (UID: \"e4ba52cf-6aac-4495-810c-8ba495c72d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"
Oct 29 23:25:37.364254 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.361117    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:37.387845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.381617    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8kzm\" (UniqueName: \"kubernetes.io/projected/e4ba52cf-6aac-4495-810c-8ba495c72d70-kube-api-access-h8kzm\") pod \"maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t\" (UID: \"e4ba52cf-6aac-4495-810c-8ba495c72d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"
Oct 29 23:25:37.476840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.476782    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"
Oct 29 23:25:37.790063 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:37.790037    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"]
Oct 29 23:25:37.790865 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:25:37.790837    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ba52cf_6aac_4495_810c_8ba495c72d70.slice/crio-51c398a0185e5de403e55fe161b0ec1becfbddd94798647bd039b9ce2f125f46 WatchSource:0}: Error finding container 51c398a0185e5de403e55fe161b0ec1becfbddd94798647bd039b9ce2f125f46: Status 404 returned error can't find the container with id 51c398a0185e5de403e55fe161b0ec1becfbddd94798647bd039b9ce2f125f46
Oct 29 23:25:38.098759 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:38.098668    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t" event={"ID":"e4ba52cf-6aac-4495-810c-8ba495c72d70","Type":"ContainerStarted","Data":"51c398a0185e5de403e55fe161b0ec1becfbddd94798647bd039b9ce2f125f46"}
Oct 29 23:25:39.021280 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:39.021250    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh"
Oct 29 23:25:39.021436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:39.021313    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg"
Oct 29 23:25:39.111416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:39.111380    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t" event={"ID":"e4ba52cf-6aac-4495-810c-8ba495c72d70","Type":"ContainerStarted","Data":"88fffa18ec8f06b605a8159f05fe565fd692960761daf8c09a89d70d61a884a4"}
Oct 29 23:25:39.112110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:39.111447    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"
Oct 29 23:25:39.156441 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:39.156378    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t" podStartSLOduration=1.775659377 podStartE2EDuration="2.156357131s" totalImagesPullingTime="380.697754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:39.155790332 +0000 UTC m=+2542.498739078" watchObservedRunningTime="2026-10-29 23:25:39.156357131 +0000 UTC m=+2542.499305877"
Oct 29 23:25:39.378844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:39.378754    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:40.689377 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:40.689345    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:40.806500 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:40.806470    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:40.924834 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:40.924788    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:41.179305 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:41.179262    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:43.192904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:43.192872    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:43.935137 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:43.935106    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:45.109394 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:45.109350    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:45.127319 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:45.127292    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"
Oct 29 23:25:45.336795 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:45.336760    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:45.465633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:45.465596    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:45.770885 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:45.770775    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:45.925969 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:45.925900    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:46.065058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:46.064891    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:46.520633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:46.520595    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:46.640549 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:46.640518    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:47.370844 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:25:47.370793    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:25:47.454572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:47.454518    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:47.570102 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:47.570071    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:47.729591 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:47.729464    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:47.870234 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:47.870122    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:48.414789 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:48.414754    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:48.531231 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:48.531190    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:49.421151 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:49.421117    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:50.532846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:50.532780    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:52.379974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:52.379936    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:54.164535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:54.164502    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:55.986413 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:55.986381    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:57.779953 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:57.779913    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:25:59.540254 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:25:59.540220    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:01.245300 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:01.245247    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:03.180734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:03.180697    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:04.874195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:04.874151    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:06.602252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:06.602184    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:08.368877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:08.368773    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:10.216387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:10.216353    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:12.687483 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:12.687448    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:15.350071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:15.350035    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:17.361552 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:17.361503    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:17.376816 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:26:17.376766    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:26:18.233928 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:18.233891    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:18.376096 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:18.376060    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:19.589283 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:19.589246    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:19.701308 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:19.701223    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:20.492993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:20.492954    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:22.790433 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:22.790372    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:24.798055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:24.798015    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:26.929032 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:26.928975    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:27.848648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:27.848607    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:29.099572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:29.099537    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:29.232588 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:29.232547    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:29.394290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:29.394258    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:29.816023 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:29.815882    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:29.986496 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:29.986461    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:30.480884 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:30.480845    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:30.618692 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:30.618655    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:31.452535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:31.452503    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:31.719045 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:31.718967    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:34.247540 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.247496    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"]
Oct 29 23:26:34.252823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.252772    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:26:34.282472 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.277233    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"]
Oct 29 23:26:34.282472 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.279029    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-45036b-serving-cert\""
Oct 29 23:26:34.298839 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.295159    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7c35813-8847-4178-9941-e6e5165078ac-maas-api-tls\") pod \"maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2\" (UID: \"d7c35813-8847-4178-9941-e6e5165078ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:26:34.298839 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.295314    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c35813-8847-4178-9941-e6e5165078ac-metrics-certs\") pod \"maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2\" (UID: \"d7c35813-8847-4178-9941-e6e5165078ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:26:34.298839 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.295484    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c35813-8847-4178-9941-e6e5165078ac-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2\" (UID: \"d7c35813-8847-4178-9941-e6e5165078ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:26:34.298839 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.295665    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn6jm\" (UniqueName: \"kubernetes.io/projected/d7c35813-8847-4178-9941-e6e5165078ac-kube-api-access-xn6jm\") pod \"maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2\" (UID: \"d7c35813-8847-4178-9941-e6e5165078ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:26:34.358510 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.358474    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:34.396551 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.396495    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c35813-8847-4178-9941-e6e5165078ac-metrics-certs\") pod \"maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2\" (UID: \"d7c35813-8847-4178-9941-e6e5165078ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:26:34.396757 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.396607    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c35813-8847-4178-9941-e6e5165078ac-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2\" (UID: \"d7c35813-8847-4178-9941-e6e5165078ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:26:34.396875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.396757    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn6jm\" (UniqueName: \"kubernetes.io/projected/d7c35813-8847-4178-9941-e6e5165078ac-kube-api-access-xn6jm\") pod \"maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2\" (UID: \"d7c35813-8847-4178-9941-e6e5165078ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:26:34.396875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.396847    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7c35813-8847-4178-9941-e6e5165078ac-maas-api-tls\") pod \"maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2\" (UID: \"d7c35813-8847-4178-9941-e6e5165078ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:26:34.397052 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:26:34.397025    5428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-serving-cert: secret "maas-api-e2e-extmod-a-45036b-serving-cert" not found
Oct 29 23:26:34.397124 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:26:34.397112    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c35813-8847-4178-9941-e6e5165078ac-maas-api-tls podName:d7c35813-8847-4178-9941-e6e5165078ac nodeName:}" failed. No retries permitted until 2026-10-29 23:26:34.897088673 +0000 UTC m=+2598.240037420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d7c35813-8847-4178-9941-e6e5165078ac-maas-api-tls") pod "maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2" (UID: "d7c35813-8847-4178-9941-e6e5165078ac") : secret "maas-api-e2e-extmod-a-45036b-serving-cert" not found
Oct 29 23:26:34.397453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.397430    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c35813-8847-4178-9941-e6e5165078ac-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2\" (UID: \"d7c35813-8847-4178-9941-e6e5165078ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:26:34.399434 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.399396    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c35813-8847-4178-9941-e6e5165078ac-metrics-certs\") pod \"maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2\" (UID: \"d7c35813-8847-4178-9941-e6e5165078ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:26:34.421236 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.421204    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn6jm\" (UniqueName: \"kubernetes.io/projected/d7c35813-8847-4178-9941-e6e5165078ac-kube-api-access-xn6jm\") pod \"maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2\" (UID: \"d7c35813-8847-4178-9941-e6e5165078ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:26:34.501556 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.501476    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:34.780503 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.780424    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:34.902771 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.902738    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7c35813-8847-4178-9941-e6e5165078ac-maas-api-tls\") pod \"maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2\" (UID: \"d7c35813-8847-4178-9941-e6e5165078ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:26:34.905381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:34.905354    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7c35813-8847-4178-9941-e6e5165078ac-maas-api-tls\") pod \"maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2\" (UID: \"d7c35813-8847-4178-9941-e6e5165078ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:26:35.168910 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:35.168877    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:26:35.480686 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:35.480660    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"]
Oct 29 23:26:35.481936 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:26:35.481907    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c35813_8847_4178_9941_e6e5165078ac.slice/crio-4eb570f8fdfc1bf590c020e0a5a896854827c2c596b2384d922e7b90b3ab88d6 WatchSource:0}: Error finding container 4eb570f8fdfc1bf590c020e0a5a896854827c2c596b2384d922e7b90b3ab88d6: Status 404 returned error can't find the container with id 4eb570f8fdfc1bf590c020e0a5a896854827c2c596b2384d922e7b90b3ab88d6
Oct 29 23:26:35.626436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:35.626401    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2" event={"ID":"d7c35813-8847-4178-9941-e6e5165078ac","Type":"ContainerStarted","Data":"4eb570f8fdfc1bf590c020e0a5a896854827c2c596b2384d922e7b90b3ab88d6"}
Oct 29 23:26:36.487634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:36.487595    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:36.640489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:36.640439    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2" event={"ID":"d7c35813-8847-4178-9941-e6e5165078ac","Type":"ContainerStarted","Data":"1dfe94f2335cdc4870db82424ebe0ac3c1dde5a02618416ddf17143369085c1c"}
Oct 29 23:26:36.640860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:36.640830    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:26:36.686492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:36.686437    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2" podStartSLOduration=2.373733059 podStartE2EDuration="2.686421077s" totalImagesPullingTime="312.688018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:36.682773993 +0000 UTC m=+2600.025722740" watchObservedRunningTime="2026-10-29 23:26:36.686421077 +0000 UTC m=+2600.029369822"
Oct 29 23:26:38.335932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:38.335901    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:40.190610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:40.190575    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:42.180164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:42.180130    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:42.655563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:42.655533    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:26:44.276007 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:44.275975    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:44.659440 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:44.659411    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:45.811450 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:45.811416    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:45.936484 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:45.936452    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:46.109042 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:46.108964    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:46.375773 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:46.375742    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:46.519644 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:46.519615    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:46.634565 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:46.634477    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:47.272571 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:47.272539    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:47.381377 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:26:47.381223    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:26:47.391109 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:47.391075    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:48.388582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:48.388549    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:52.172945 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:52.172909    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:52.502151 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:52.502063    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:53.172887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:53.172852    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:54.321391 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:54.321352    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:54.486753 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:54.486711    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:54.720625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:54.720596    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:56.375231 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:56.375197    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:58.191713 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:58.191640    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:26:59.999897 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:26:59.999855    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:01.800476 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:01.800444    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:03.571737 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:03.571701    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:03.828038 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:03.827966    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:03.949406 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:03.949374    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:04.453648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:04.453612    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:05.012217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:05.012176    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:05.165252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:05.165215    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:05.298366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:05.298280    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:05.437147 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:05.437098    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:05.598637 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:05.598550    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:05.795928 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:05.795892    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:05.918704 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:05.918669    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:06.698935 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:06.698899    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:06.820741 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:06.820702    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:07.332642 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:07.332610    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:07.528104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:07.528073    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:09.287837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:09.287789    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:10.538688 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:10.538658    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:10.708794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:10.708762    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:10.831989 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:10.831913    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:11.148016 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:11.147986    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:13.296448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:13.296410    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:14.038158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:14.038121    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:14.160711 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:14.160637    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:14.289692 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:14.289594    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:15.356917 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:15.356874    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:17.376464 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:17.376418    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:27:17.421729 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:17.421697    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:17.550196 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:17.550159    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:17.872852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:17.872811    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:18.073594 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:18.073566    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:18.743689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:18.743650    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:20.567819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:20.567759    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:20.895041 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:20.895009    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:21.065220 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:21.065121    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:21.247037 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:21.246950    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:21.371486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:21.371451    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:21.601687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:21.601601    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:21.749573 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:21.749543    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:21.992692 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:21.992655    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:22.551817 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:22.551777    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:22.689149 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:22.689117    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:22.803001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:22.802913    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:23.743212 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:23.743176    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:23.991778 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:23.991700    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"]
Oct 29 23:27:23.997446 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:23.997380    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"
Oct 29 23:27:24.017920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.017892    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"]
Oct 29 23:27:24.024705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.024675    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c5ef2f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c5ef2f\""
Oct 29 23:27:24.024887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.024688    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c5ef2f\"/\"routing-overlay\""
Oct 29 23:27:24.024887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.024705    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c5ef2f\"/\"kube-root-ca.crt\""
Oct 29 23:27:24.024887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.024759    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c5ef2f\"/\"openshift-service-ca.crt\""
Oct 29 23:27:24.031083 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.031061    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c5ef2f\"/\"agc-cred-fd8530b2a258\""
Oct 29 23:27:24.031209 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.031098    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c5ef2f\"/\"payload-processing-external-model-e2e--ca6efe92-dockercfg-xw8pp\""
Oct 29 23:27:24.080241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.080207    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/432a4527-a0fc-41dc-b131-4843d3d7e673-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j\" (UID: \"432a4527-a0fc-41dc-b131-4843d3d7e673\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"
Oct 29 23:27:24.080432 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.080357    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j\" (UID: \"432a4527-a0fc-41dc-b131-4843d3d7e673\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"
Oct 29 23:27:24.080613 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.080585    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432a4527-a0fc-41dc-b131-4843d3d7e673-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j\" (UID: \"432a4527-a0fc-41dc-b131-4843d3d7e673\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"
Oct 29 23:27:24.080752 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.080736    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jww95\" (UniqueName: \"kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-kube-api-access-jww95\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j\" (UID: \"432a4527-a0fc-41dc-b131-4843d3d7e673\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"
Oct 29 23:27:24.183467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.183430    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432a4527-a0fc-41dc-b131-4843d3d7e673-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j\" (UID: \"432a4527-a0fc-41dc-b131-4843d3d7e673\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"
Oct 29 23:27:24.183667 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.183566    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jww95\" (UniqueName: \"kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-kube-api-access-jww95\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j\" (UID: \"432a4527-a0fc-41dc-b131-4843d3d7e673\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"
Oct 29 23:27:24.183667 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.183601    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/432a4527-a0fc-41dc-b131-4843d3d7e673-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j\" (UID: \"432a4527-a0fc-41dc-b131-4843d3d7e673\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"
Oct 29 23:27:24.183848 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.183694    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j\" (UID: \"432a4527-a0fc-41dc-b131-4843d3d7e673\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"
Oct 29 23:27:24.190872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.188000    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/432a4527-a0fc-41dc-b131-4843d3d7e673-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j\" (UID: \"432a4527-a0fc-41dc-b131-4843d3d7e673\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"
Oct 29 23:27:24.193550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.193522    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432a4527-a0fc-41dc-b131-4843d3d7e673-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j\" (UID: \"432a4527-a0fc-41dc-b131-4843d3d7e673\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"
Oct 29 23:27:24.200695 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.200642    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j\" (UID: \"432a4527-a0fc-41dc-b131-4843d3d7e673\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"
Oct 29 23:27:24.216342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.216310    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jww95\" (UniqueName: \"kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-kube-api-access-jww95\") pod \"payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j\" (UID: \"432a4527-a0fc-41dc-b131-4843d3d7e673\") " pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"
Oct 29 23:27:24.313025 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.312940    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"
Oct 29 23:27:24.625311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:24.625284    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"]
Oct 29 23:27:24.626519 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:27:24.626481    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432a4527_a0fc_41dc_b131_4843d3d7e673.slice/crio-0a19d667fe89ed5c14d0ac34419a847d8e2edfc3033c33daed6a4de56d5a67c6 WatchSource:0}: Error finding container 0a19d667fe89ed5c14d0ac34419a847d8e2edfc3033c33daed6a4de56d5a67c6: Status 404 returned error can't find the container with id 0a19d667fe89ed5c14d0ac34419a847d8e2edfc3033c33daed6a4de56d5a67c6
Oct 29 23:27:25.052560 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:25.052472    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:25.230849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:25.229947    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j" event={"ID":"432a4527-a0fc-41dc-b131-4843d3d7e673","Type":"ContainerStarted","Data":"76d77e3993bbcd67e6913f746043f03f44ad51e7d94bbcc376edcb51818a7348"}
Oct 29 23:27:25.230849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:25.230044    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j" event={"ID":"432a4527-a0fc-41dc-b131-4843d3d7e673","Type":"ContainerStarted","Data":"0a19d667fe89ed5c14d0ac34419a847d8e2edfc3033c33daed6a4de56d5a67c6"}
Oct 29 23:27:25.230849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:25.230046    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:25.230849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:25.230124    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"
Oct 29 23:27:25.275131 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:25.275079    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j" podStartSLOduration=2.275065746 podStartE2EDuration="2.275065746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:25.271010898 +0000 UTC m=+2648.613959643" watchObservedRunningTime="2026-10-29 23:27:25.275065746 +0000 UTC m=+2648.618014494"
Oct 29 23:27:25.445458 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:25.445414    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:25.569856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:25.569155    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:26.328002 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:26.327964    5428 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c5ef2f/agc-cred-fd8530b2a258: secret "agc-cred-fd8530b2a258" not found
Oct 29 23:27:26.328002 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:26.327994    5428 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j: secret "agc-cred-fd8530b2a258" not found
Oct 29 23:27:26.328483 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:26.328069    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials podName:432a4527-a0fc-41dc-b131-4843d3d7e673 nodeName:}" failed. No retries permitted until 2026-10-29 23:27:26.828047712 +0000 UTC m=+2650.170996445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j" (UID: "432a4527-a0fc-41dc-b131-4843d3d7e673") : secret "agc-cred-fd8530b2a258" not found
Oct 29 23:27:26.836884 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:26.836850    5428 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c5ef2f/agc-cred-fd8530b2a258: secret "agc-cred-fd8530b2a258" not found
Oct 29 23:27:26.836884 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:26.836874    5428 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j: secret "agc-cred-fd8530b2a258" not found
Oct 29 23:27:26.837123 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:26.836922    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials podName:432a4527-a0fc-41dc-b131-4843d3d7e673 nodeName:}" failed. No retries permitted until 2026-10-29 23:27:27.836907576 +0000 UTC m=+2651.179856299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j" (UID: "432a4527-a0fc-41dc-b131-4843d3d7e673") : secret "agc-cred-fd8530b2a258" not found
Oct 29 23:27:27.488343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:27.488310    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:27.852609 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:27.852527    5428 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c5ef2f/agc-cred-fd8530b2a258: secret "agc-cred-fd8530b2a258" not found
Oct 29 23:27:27.852609 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:27.852555    5428 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j: secret "agc-cred-fd8530b2a258" not found
Oct 29 23:27:27.852609 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:27.852613    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials podName:432a4527-a0fc-41dc-b131-4843d3d7e673 nodeName:}" failed. No retries permitted until 2026-10-29 23:27:29.852599225 +0000 UTC m=+2653.195547961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j" (UID: "432a4527-a0fc-41dc-b131-4843d3d7e673") : secret "agc-cred-fd8530b2a258" not found
Oct 29 23:27:29.273950 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:29.273902    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:29.882766 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:29.882732    5428 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c5ef2f/agc-cred-fd8530b2a258: secret "agc-cred-fd8530b2a258" not found
Oct 29 23:27:29.882766 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:29.882764    5428 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j: secret "agc-cred-fd8530b2a258" not found
Oct 29 23:27:29.883119 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:29.882859    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials podName:432a4527-a0fc-41dc-b131-4843d3d7e673 nodeName:}" failed. No retries permitted until 2026-10-29 23:27:33.88283749 +0000 UTC m=+2657.225786219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j" (UID: "432a4527-a0fc-41dc-b131-4843d3d7e673") : secret "agc-cred-fd8530b2a258" not found
Oct 29 23:27:30.644979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:30.644944    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:31.007683 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:31.007592    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:31.303918 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:31.303843    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:32.874975 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:32.874946    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:33.561168 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.561136    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"]
Oct 29 23:27:33.567753 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.567728    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.582529 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.582507    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 29 23:27:33.582529 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.582506    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 29 23:27:33.582690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.582534    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7dsjj\""
Oct 29 23:27:33.582690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.582518    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 29 23:27:33.600104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.600076    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"]
Oct 29 23:27:33.639892 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.639850    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.640078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.640045    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh66c\" (UniqueName: \"kubernetes.io/projected/43df95b6-fe67-4edd-aeea-ff29d29c94c2-kube-api-access-zh66c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.640173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.640106    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.640288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.640268    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.640435 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.640401    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.640545 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.640529    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/43df95b6-fe67-4edd-aeea-ff29d29c94c2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.640642 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.640627    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.742651 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.742618    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.742825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.742701    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/43df95b6-fe67-4edd-aeea-ff29d29c94c2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.742825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.742756    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.742955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.742939    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.743013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.742996    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh66c\" (UniqueName: \"kubernetes.io/projected/43df95b6-fe67-4edd-aeea-ff29d29c94c2-kube-api-access-zh66c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.743078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.743026    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.743178 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.743159    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.743975 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.743945    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.745373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.745341    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/43df95b6-fe67-4edd-aeea-ff29d29c94c2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.745467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.745416    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.745573 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.745550    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.745623 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.745603    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.765850 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.765793    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.766691 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.766668    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh66c\" (UniqueName: \"kubernetes.io/projected/43df95b6-fe67-4edd-aeea-ff29d29c94c2-kube-api-access-zh66c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cxw46\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.882168 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:33.882140    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:33.946833 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:33.946786    5428 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c5ef2f/agc-cred-fd8530b2a258: secret "agc-cred-fd8530b2a258" not found
Oct 29 23:27:33.946954 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:33.946837    5428 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j: secret "agc-cred-fd8530b2a258" not found
Oct 29 23:27:33.947024 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:33.946966    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials podName:432a4527-a0fc-41dc-b131-4843d3d7e673 nodeName:}" failed. No retries permitted until 2026-10-29 23:27:41.946907768 +0000 UTC m=+2665.289856499 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j" (UID: "432a4527-a0fc-41dc-b131-4843d3d7e673") : secret "agc-cred-fd8530b2a258" not found
Oct 29 23:27:34.211720 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:27:34.211690    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43df95b6_fe67_4edd_aeea_ff29d29c94c2.slice/crio-3a0b1734d413f599cdae7d6cf1e155fd0e567608de99152f0297540f80f1c7cd WatchSource:0}: Error finding container 3a0b1734d413f599cdae7d6cf1e155fd0e567608de99152f0297540f80f1c7cd: Status 404 returned error can't find the container with id 3a0b1734d413f599cdae7d6cf1e155fd0e567608de99152f0297540f80f1c7cd
Oct 29 23:27:34.214832 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:34.214791    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"]
Oct 29 23:27:34.214940 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:34.214909    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:27:34.214984 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:34.214972    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:27:34.311834 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:34.311758    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46" event={"ID":"43df95b6-fe67-4edd-aeea-ff29d29c94c2","Type":"ContainerStarted","Data":"8d086784ef382515f0d9a163d5af9cf9528550b9e7f2f1e84b1b051af6c44f0d"}
Oct 29 23:27:34.312019 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:34.312001    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:34.312086 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:34.312032    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46" event={"ID":"43df95b6-fe67-4edd-aeea-ff29d29c94c2","Type":"ContainerStarted","Data":"3a0b1734d413f599cdae7d6cf1e155fd0e567608de99152f0297540f80f1c7cd"}
Oct 29 23:27:34.364302 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:34.364242    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46" podStartSLOduration=1.3642281010000001 podStartE2EDuration="1.364228101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:34.360361711 +0000 UTC m=+2657.703310481" watchObservedRunningTime="2026-10-29 23:27:34.364228101 +0000 UTC m=+2657.707176845"
Oct 29 23:27:34.652001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:34.651960    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:35.330010 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:35.329973    5428 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-cxw46 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:27:35.330493 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:35.330047    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46" podUID="43df95b6-fe67-4edd-aeea-ff29d29c94c2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:27:36.241906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:36.241876    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"
Oct 29 23:27:36.517900 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:36.517772    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:38.328401 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:38.328374    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:27:38.361846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:38.361770    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:38.444882 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:38.444842    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"]
Oct 29 23:27:39.561244 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.561211    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"]
Oct 29 23:27:39.566483 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.566461    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.598586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.598553    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"]
Oct 29 23:27:39.606373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.605471    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"]
Oct 29 23:27:39.618199 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.615582    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.634234 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.634209    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"]
Oct 29 23:27:39.649857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.649827    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.649996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.649877    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.650060 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.649983    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkqwf\" (UniqueName: \"kubernetes.io/projected/bf911873-fc76-4358-8c69-b6d644701bfd-kube-api-access-fkqwf\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.650130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.650079    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fcxw\" (UniqueName: \"kubernetes.io/projected/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-kube-api-access-7fcxw\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.650190 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.650124    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.650243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.650195    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf911873-fc76-4358-8c69-b6d644701bfd-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.650326 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.650308    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.650427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.650410    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.650497 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.650443    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf911873-fc76-4358-8c69-b6d644701bfd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.650567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.650547    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.650635 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.650612    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.650718 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.650696    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf911873-fc76-4358-8c69-b6d644701bfd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.650857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.650841    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.650940 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.650877    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.651025 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.651004    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.651083 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.651038    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.651083 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.651066    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.651168 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.651136    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.753027 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.752990    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.753219 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753037    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.753219 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753077    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.753219 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753093    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.753219 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753111    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.753219 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753132    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.753219 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753202    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.753219 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753223    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.753638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753239    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkqwf\" (UniqueName: \"kubernetes.io/projected/bf911873-fc76-4358-8c69-b6d644701bfd-kube-api-access-fkqwf\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.753638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753271    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fcxw\" (UniqueName: \"kubernetes.io/projected/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-kube-api-access-7fcxw\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.753638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753296    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.753638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753331    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf911873-fc76-4358-8c69-b6d644701bfd-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.753638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753387    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.753638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753465    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.753638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753494    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf911873-fc76-4358-8c69-b6d644701bfd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.753638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753521    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.753638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753527    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.753638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753591    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.753638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753603    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.754396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753685    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf911873-fc76-4358-8c69-b6d644701bfd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.754396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753740    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.754396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753786    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.754396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.753489    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.754396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.754012    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.754396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.754073    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.754396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.754092    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.754396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.754170    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.754765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.754622    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf911873-fc76-4358-8c69-b6d644701bfd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.756509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.756476    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.756652 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.756523    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.756729 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.756674    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf911873-fc76-4358-8c69-b6d644701bfd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.756784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.756725    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.783738 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.783714    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkqwf\" (UniqueName: \"kubernetes.io/projected/bf911873-fc76-4358-8c69-b6d644701bfd-kube-api-access-fkqwf\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.784078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.784056    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf911873-fc76-4358-8c69-b6d644701bfd-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-vdxdj\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:39.786928 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.786900    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.788714 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.788690    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fcxw\" (UniqueName: \"kubernetes.io/projected/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-kube-api-access-7fcxw\") pod \"maas-default-gateway-openshift-default-55fbd65575-5fg8j\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.883747 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.883714    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:39.947158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:39.947126    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:40.211409 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:40.211383    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"]
Oct 29 23:27:40.211661 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:27:40.211635    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6c8bc1_5531_4ae7_9660_67ee6263b34e.slice/crio-a7fb930c520e8069614e517da54eced19728cf07b84456142e6dc918e1a6ecb9 WatchSource:0}: Error finding container a7fb930c520e8069614e517da54eced19728cf07b84456142e6dc918e1a6ecb9: Status 404 returned error can't find the container with id a7fb930c520e8069614e517da54eced19728cf07b84456142e6dc918e1a6ecb9
Oct 29 23:27:40.214876 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:40.214846    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:27:40.214961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:40.214907    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:27:40.214961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:40.214936    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:27:40.219225 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:40.219197    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:40.287776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:40.285761    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"]
Oct 29 23:27:40.290887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:40.290853    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:27:40.291015 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:40.290923    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:27:40.291015 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:40.290954    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:27:40.376679 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:40.376641    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j" event={"ID":"4c6c8bc1-5531-4ae7-9660-67ee6263b34e","Type":"ContainerStarted","Data":"aa37b38e4737888ccc6e5c62449060fc8e09bf20d5de7c2f64187cdd147dccac"}
Oct 29 23:27:40.376882 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:40.376704    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j" event={"ID":"4c6c8bc1-5531-4ae7-9660-67ee6263b34e","Type":"ContainerStarted","Data":"a7fb930c520e8069614e517da54eced19728cf07b84456142e6dc918e1a6ecb9"}
Oct 29 23:27:40.378893 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:40.378861    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj" event={"ID":"bf911873-fc76-4358-8c69-b6d644701bfd","Type":"ContainerStarted","Data":"44d9bf05926f029f4b73deea3fb59c1f7e5164671b6fbd22d55eb6b92721b900"}
Oct 29 23:27:40.431971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:40.431922    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j" podStartSLOduration=1.431906785 podStartE2EDuration="1.431906785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:40.427389916 +0000 UTC m=+2663.770338689" watchObservedRunningTime="2026-10-29 23:27:40.431906785 +0000 UTC m=+2663.774855529"
Oct 29 23:27:40.884236 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:40.884208    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:41.421559 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:41.421521    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj" event={"ID":"bf911873-fc76-4358-8c69-b6d644701bfd","Type":"ContainerStarted","Data":"45bef36ee2dcfda2fd5a7e0b248ca63b252174ea434cb29274adc647137abe81"}
Oct 29 23:27:41.515822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:41.508967    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj" podStartSLOduration=2.508944579 podStartE2EDuration="2.508944579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:41.487364074 +0000 UTC m=+2664.830312820" watchObservedRunningTime="2026-10-29 23:27:41.508944579 +0000 UTC m=+2664.851893323"
Oct 29 23:27:41.832874 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:41.832488    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:41.993827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:41.992635    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:42.007445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:41.994890    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-5fg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 29 23:27:42.007825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:42.007760    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j" podUID="4c6c8bc1-5531-4ae7-9660-67ee6263b34e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 29 23:27:42.017693 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:42.017651    5428 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c5ef2f/agc-cred-fd8530b2a258: secret "agc-cred-fd8530b2a258" not found
Oct 29 23:27:42.017892 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:42.017876    5428 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j: secret "agc-cred-fd8530b2a258" not found
Oct 29 23:27:42.018067 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:42.018053    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials podName:432a4527-a0fc-41dc-b131-4843d3d7e673 nodeName:}" failed. No retries permitted until 2026-10-29 23:27:58.018031075 +0000 UTC m=+2681.360979802 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j" (UID: "432a4527-a0fc-41dc-b131-4843d3d7e673") : secret "agc-cred-fd8530b2a258" not found
Oct 29 23:27:42.022313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:42.022285    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:42.029741 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:42.029719    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-vdxdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 29 23:27:42.031482 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:42.031439    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj" podUID="bf911873-fc76-4358-8c69-b6d644701bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 29 23:27:42.891827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:42.889138    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-5fg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 29 23:27:42.891827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:42.889213    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j" podUID="4c6c8bc1-5531-4ae7-9660-67ee6263b34e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 29 23:27:42.947895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:42.947864    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-vdxdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 29 23:27:42.948071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:42.947912    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj" podUID="bf911873-fc76-4358-8c69-b6d644701bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 29 23:27:43.445840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:43.445782    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j" podUID="432a4527-a0fc-41dc-b131-4843d3d7e673" containerName="payload-processing" containerID="cri-o://76d77e3993bbcd67e6913f746043f03f44ad51e7d94bbcc376edcb51818a7348" gracePeriod=25
Oct 29 23:27:43.537736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:43.537695    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:43.902825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:43.901939    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-5fg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 29 23:27:43.902825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:43.902000    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j" podUID="4c6c8bc1-5531-4ae7-9660-67ee6263b34e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 29 23:27:43.947728 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:43.947700    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-vdxdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 29 23:27:43.947863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:43.947746    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj" podUID="bf911873-fc76-4358-8c69-b6d644701bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 29 23:27:43.974849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:43.958635    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"
Oct 29 23:27:44.028031 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.027947    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-kube-api-access-jww95\" (UniqueName: \"kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-kube-api-access-jww95\") pod \"432a4527-a0fc-41dc-b131-4843d3d7e673\" (UID: \"432a4527-a0fc-41dc-b131-4843d3d7e673\") "
Oct 29 23:27:44.028166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.028028    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432a4527-a0fc-41dc-b131-4843d3d7e673-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/432a4527-a0fc-41dc-b131-4843d3d7e673-routing-overlay-volume\") pod \"432a4527-a0fc-41dc-b131-4843d3d7e673\" (UID: \"432a4527-a0fc-41dc-b131-4843d3d7e673\") "
Oct 29 23:27:44.028221 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.028210    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials\" (UniqueName: \"kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials\") pod \"432a4527-a0fc-41dc-b131-4843d3d7e673\" (UID: \"432a4527-a0fc-41dc-b131-4843d3d7e673\") "
Oct 29 23:27:44.028259 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.028243    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432a4527-a0fc-41dc-b131-4843d3d7e673-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432a4527-a0fc-41dc-b131-4843d3d7e673-plugins-config-volume\") pod \"432a4527-a0fc-41dc-b131-4843d3d7e673\" (UID: \"432a4527-a0fc-41dc-b131-4843d3d7e673\") "
Oct 29 23:27:44.029279 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.029236    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432a4527-a0fc-41dc-b131-4843d3d7e673-routing-overlay-volume" pod "432a4527-a0fc-41dc-b131-4843d3d7e673" (UID: "432a4527-a0fc-41dc-b131-4843d3d7e673"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:27:44.029279 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.029268    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432a4527-a0fc-41dc-b131-4843d3d7e673-plugins-config-volume" pod "432a4527-a0fc-41dc-b131-4843d3d7e673" (UID: "432a4527-a0fc-41dc-b131-4843d3d7e673"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:27:44.032069 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.032040    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials" pod "432a4527-a0fc-41dc-b131-4843d3d7e673" (UID: "432a4527-a0fc-41dc-b131-4843d3d7e673"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:27:44.035011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.034990    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-kube-api-access-jww95" pod "432a4527-a0fc-41dc-b131-4843d3d7e673" (UID: "432a4527-a0fc-41dc-b131-4843d3d7e673"). InnerVolumeSpecName "kube-api-access-jww95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:27:44.132921 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.132887    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jww95\" (UniqueName: \"kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-kube-api-access-jww95\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:27:44.132921 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.132912    5428 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/432a4527-a0fc-41dc-b131-4843d3d7e673-routing-overlay-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:27:44.132921 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.132925    5428 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/432a4527-a0fc-41dc-b131-4843d3d7e673-provider-credentials\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:27:44.133222 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.132935    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432a4527-a0fc-41dc-b131-4843d3d7e673-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:27:44.527680 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.527638    5428 generic.go:356] "Generic (PLEG): container finished" podID="432a4527-a0fc-41dc-b131-4843d3d7e673" containerID="76d77e3993bbcd67e6913f746043f03f44ad51e7d94bbcc376edcb51818a7348" exitCode=0
Oct 29 23:27:44.528143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.527688    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"
Oct 29 23:27:44.528143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.527717    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j" event={"ID":"432a4527-a0fc-41dc-b131-4843d3d7e673","Type":"ContainerDied","Data":"76d77e3993bbcd67e6913f746043f03f44ad51e7d94bbcc376edcb51818a7348"}
Oct 29 23:27:44.528143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.527773    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j" event={"ID":"432a4527-a0fc-41dc-b131-4843d3d7e673","Type":"ContainerDied","Data":"0a19d667fe89ed5c14d0ac34419a847d8e2edfc3033c33daed6a4de56d5a67c6"}
Oct 29 23:27:44.528143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.527822    5428 scope.go:122] "RemoveContainer" containerID="76d77e3993bbcd67e6913f746043f03f44ad51e7d94bbcc376edcb51818a7348"
Oct 29 23:27:44.540366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.540347    5428 scope.go:122] "RemoveContainer" containerID="76d77e3993bbcd67e6913f746043f03f44ad51e7d94bbcc376edcb51818a7348"
Oct 29 23:27:44.540743 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:44.540722    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d77e3993bbcd67e6913f746043f03f44ad51e7d94bbcc376edcb51818a7348\": container with ID starting with 76d77e3993bbcd67e6913f746043f03f44ad51e7d94bbcc376edcb51818a7348 not found: ID does not exist" containerID="76d77e3993bbcd67e6913f746043f03f44ad51e7d94bbcc376edcb51818a7348"
Oct 29 23:27:44.540833 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.540751    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d77e3993bbcd67e6913f746043f03f44ad51e7d94bbcc376edcb51818a7348"} err="failed to get container status \"76d77e3993bbcd67e6913f746043f03f44ad51e7d94bbcc376edcb51818a7348\": rpc error: code = NotFound desc = could not find container \"76d77e3993bbcd67e6913f746043f03f44ad51e7d94bbcc376edcb51818a7348\": container with ID starting with 76d77e3993bbcd67e6913f746043f03f44ad51e7d94bbcc376edcb51818a7348 not found: ID does not exist"
Oct 29 23:27:44.615921 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.615880    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"]
Oct 29 23:27:44.618029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.618003    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c5ef2f/payload-processing-external-model-e2e-extmod-a-c5ef2f-5bf6bcl8j"]
Oct 29 23:27:44.817492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.817109    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432a4527-a0fc-41dc-b131-4843d3d7e673" path="/var/lib/kubelet/pods/432a4527-a0fc-41dc-b131-4843d3d7e673/volumes"
Oct 29 23:27:44.887831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.886912    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-5fg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 29 23:27:44.887831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.886988    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j" podUID="4c6c8bc1-5531-4ae7-9660-67ee6263b34e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 29 23:27:44.948204 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.948171    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-vdxdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 29 23:27:44.948389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:44.948233    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj" podUID="bf911873-fc76-4358-8c69-b6d644701bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 29 23:27:45.896775 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:45.896732    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-5fg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 29 23:27:45.908939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:45.908898    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j" podUID="4c6c8bc1-5531-4ae7-9660-67ee6263b34e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 29 23:27:45.948411 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:45.948381    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-vdxdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 29 23:27:45.948561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:45.948441    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj" podUID="bf911873-fc76-4358-8c69-b6d644701bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 29 23:27:46.587171 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:46.587131    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:46.884558 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:46.884513    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-5fg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 29 23:27:46.884759 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:46.884591    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j" podUID="4c6c8bc1-5531-4ae7-9660-67ee6263b34e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 29 23:27:46.948202 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:46.948160    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-vdxdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 29 23:27:46.948723 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:46.948243    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj" podUID="bf911873-fc76-4358-8c69-b6d644701bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 29 23:27:47.371149 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:27:47.371107    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:27:47.884678 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:47.884637    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-5fg8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Oct 29 23:27:47.884913 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:47.884722    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j" podUID="4c6c8bc1-5531-4ae7-9660-67ee6263b34e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Oct 29 23:27:47.948016 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:47.947971    5428 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-vdxdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 29 23:27:47.948208 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:47.948025    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj" podUID="bf911873-fc76-4358-8c69-b6d644701bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 29 23:27:48.316916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:48.316782    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:48.810371 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:48.810328    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:49.207847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:49.207819    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:49.208270 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:49.208246    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:49.208973 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:49.208948    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:27:49.229731 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:49.229702    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:49.230070 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:49.230019    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:49.231079 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:49.231063    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:27:50.848999 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:50.848965    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:52.830626 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:52.830593    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:54.054330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:54.054294    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:54.165015 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:54.164982    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:55.178778 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:55.178742    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:55.300167 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:55.300133    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:55.420834 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:55.420782    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:55.977737 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:55.977703    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:56.421857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:56.421819    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:56.546591 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:56.546546    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:57.736510 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:57.736473    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:27:59.096296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:27:59.096251    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:00.984106 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:00.984076    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:02.930585 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:02.930531    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:03.043581 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:03.043543    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:04.307656 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:04.307620    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:04.507706 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:04.507676    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:04.630597 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:04.630568    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:04.936594 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:04.936494    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:05.431507 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:05.431457    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:05.579159 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:05.579126    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:05.693470 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:05.693383    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:07.557571 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:07.557538    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:08.391750 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:08.391710    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:08.866829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:08.866640    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"]
Oct 29 23:28:08.867313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:08.866987    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8" podUID="5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" containerName="istio-proxy" containerID="cri-o://36c14d83dd3bacd8553d0b4a458b618fb9a58b21b7028e03a267357f1882e8bc" gracePeriod=30
Oct 29 23:28:09.305855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:09.305761    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:09.675075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:09.675039    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:10.182229 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:10.182171    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:10.452442 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:10.452335    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:11.419646 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:11.419605    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:11.614486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:11.614449    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:13.479402 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:13.479365    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:13.597050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:13.597010    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:28:13.597384 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:13.597350    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8" podUID="5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:28:14.234357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.234324    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:28:14.309070 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.309028    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-workload-certs\") pod \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") "
Oct 29 23:28:14.309253 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.309105    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-podinfo\") pod \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") "
Oct 29 23:28:14.309253 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.309139    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-kube-api-access-l598v\" (UniqueName: \"kubernetes.io/projected/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-kube-api-access-l598v\") pod \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") "
Oct 29 23:28:14.309253 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.309199    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-credential-socket\") pod \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") "
Oct 29 23:28:14.309417 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.309257    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-token\" (UniqueName: \"kubernetes.io/projected/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-token\") pod \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") "
Oct 29 23:28:14.309417 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.309305    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-data\") pod \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") "
Oct 29 23:28:14.309417 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.309333    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-workload-certs" pod "5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" (UID: "5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:14.309652 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.309621    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istiod-ca-cert\") pod \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") "
Oct 29 23:28:14.309769 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.309740    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-workload-socket\") pod \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") "
Oct 29 23:28:14.309877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.309766    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-credential-socket" pod "5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" (UID: "5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:14.309877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.309824    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-envoy\") pod \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\" (UID: \"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6\") "
Oct 29 23:28:14.310001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.309917    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-data" pod "5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" (UID: "5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:14.310062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.310013    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-workload-socket" pod "5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" (UID: "5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:14.310140 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.310083    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istiod-ca-cert" pod "5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" (UID: "5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:28:14.310649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.310621    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:28:14.310761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.310653    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:28:14.310761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.310669    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:28:14.310761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.310683    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:28:14.310761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.310696    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:28:14.312207 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.312178    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-kube-api-access-l598v" pod "5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" (UID: "5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6"). InnerVolumeSpecName "kube-api-access-l598v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:14.313152 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.313094    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-envoy" pod "5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" (UID: "5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:28:14.314867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.314845    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-podinfo" pod "5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" (UID: "5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:28:14.315213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.315160    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-token" pod "5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" (UID: "5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:28:14.412991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.412953    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:28:14.412991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.412993    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:28:14.413207 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.413008    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l598v\" (UniqueName: \"kubernetes.io/projected/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-kube-api-access-l598v\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:28:14.413207 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.413023    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:28:14.868295 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.868255    5428 generic.go:356] "Generic (PLEG): container finished" podID="5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" containerID="36c14d83dd3bacd8553d0b4a458b618fb9a58b21b7028e03a267357f1882e8bc" exitCode=0
Oct 29 23:28:14.868766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.868306    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"
Oct 29 23:28:14.868766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.868343    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8" event={"ID":"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6","Type":"ContainerDied","Data":"36c14d83dd3bacd8553d0b4a458b618fb9a58b21b7028e03a267357f1882e8bc"}
Oct 29 23:28:14.868766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.868410    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8" event={"ID":"5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6","Type":"ContainerDied","Data":"ba6185ac36c0b2709efffc885125b03ec291c93a917631d3ce0345ebeb011470"}
Oct 29 23:28:14.868766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.868439    5428 scope.go:122] "RemoveContainer" containerID="36c14d83dd3bacd8553d0b4a458b618fb9a58b21b7028e03a267357f1882e8bc"
Oct 29 23:28:14.880908 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.880882    5428 scope.go:122] "RemoveContainer" containerID="36c14d83dd3bacd8553d0b4a458b618fb9a58b21b7028e03a267357f1882e8bc"
Oct 29 23:28:14.881215 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:28:14.881188    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c14d83dd3bacd8553d0b4a458b618fb9a58b21b7028e03a267357f1882e8bc\": container with ID starting with 36c14d83dd3bacd8553d0b4a458b618fb9a58b21b7028e03a267357f1882e8bc not found: ID does not exist" containerID="36c14d83dd3bacd8553d0b4a458b618fb9a58b21b7028e03a267357f1882e8bc"
Oct 29 23:28:14.881294 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.881225    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c14d83dd3bacd8553d0b4a458b618fb9a58b21b7028e03a267357f1882e8bc"} err="failed to get container status \"36c14d83dd3bacd8553d0b4a458b618fb9a58b21b7028e03a267357f1882e8bc\": rpc error: code = NotFound desc = could not find container \"36c14d83dd3bacd8553d0b4a458b618fb9a58b21b7028e03a267357f1882e8bc\": container with ID starting with 36c14d83dd3bacd8553d0b4a458b618fb9a58b21b7028e03a267357f1882e8bc not found: ID does not exist"
Oct 29 23:28:14.923490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.923461    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"]
Oct 29 23:28:14.935464 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:14.935434    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b4b5e3-openshift-default-9bf4644c7-4wds8"]
Oct 29 23:28:16.458114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:16.458082    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:16.794826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:16.794727    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" path="/var/lib/kubelet/pods/5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6/volumes"
Oct 29 23:28:17.394692 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:28:17.394646    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:28:18.913705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:18.913668    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:20.028019 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:20.027948    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:20.148620 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:20.148575    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:21.436075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:21.436040    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:23.335078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:23.335045    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:24.377999 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:24.377932    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:24.511205 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:24.511172    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:25.469929 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:25.469892    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:26.016489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:26.016399    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:26.141525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:26.141491    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:27.596381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:27.596342    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:29.428321 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:29.428284    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:30.774240 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:30.774203    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:31.376882 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:31.376846    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:31.487611 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:31.487580    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:33.093472 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:33.093431    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:34.957254 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:34.957164    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:36.669579 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:36.669549    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:38.369361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:38.369326    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:40.193187 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:40.193155    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:40.329421 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:40.329391    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:41.906647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:41.906612    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:43.712642 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:43.712609    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:45.575677 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:45.575643    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:47.369507 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:28:47.369469    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:28:47.372650 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:47.372625    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:49.195094 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:49.195064    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:50.985542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:50.985504    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:52.812396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:52.812364    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:54.536937 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:54.536896    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:56.322230 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:56.322196    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:58.070345 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:58.070311    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:28:59.757172 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:28:59.757137    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:01.569614 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:01.569585    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:03.304780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:03.304748    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:05.795521 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:05.795466    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:07.686123 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:07.686087    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:09.582685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:09.582649    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:09.752213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:09.752175    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:09.873301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:09.873271    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:11.323862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:11.323789    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:12.987000 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:12.986961    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:14.884464 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:14.884429    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:16.678003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:16.677965    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:17.012636 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:17.012533    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:17.392362 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:29:17.392035    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:29:17.883354 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:17.883320    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:18.005960 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:18.005930    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:19.146839 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:19.146791    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:19.658089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:19.658060    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:21.895201 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:21.895167    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:23.684856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:23.684780    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:25.262630 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:25.262601    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:25.391546 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:25.391497    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:25.766941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:25.766909    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:26.115178 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:26.115141    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:27.553408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:27.553375    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:27.685971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:27.685933    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:28.286411 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:28.286381    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:28.430292 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:28.430250    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:28.548196 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:28.548111    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:30.485325 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:30.485292    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:30.885744 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:30.885714    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:31.549949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:31.549890    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:32.265013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:32.264981    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:34.069980 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:34.069946    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:35.000115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:35.000080    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:35.145042 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:35.145009    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:36.136488 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:36.136430    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:36.248037 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:36.248001    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:37.313170 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:37.313139    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:39.180509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:39.180478    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:41.060670 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:41.060637    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:41.437261 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:41.437225    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:41.556728 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:41.556693    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:43.824648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:43.824609    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:45.719135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:45.719101    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:47.369277 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:29:47.369237    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:29:47.569122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:47.569085    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:49.240357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:49.240322    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:49.384573 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:49.384531    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:49.585253 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:49.585131    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:49.880582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:49.880544    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:51.885311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:51.885278    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:52.064817 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:52.064781    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:53.706297 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:53.706258    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:55.501400 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:55.501363    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:56.155687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:56.155651    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:56.279151 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:56.279122    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:56.400096 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:56.400061    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:56.520533 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:56.520453    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:56.632167 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:56.632125    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:56.762330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:56.762297    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:57.379677 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:57.379644    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:57.608554 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:57.608510    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:57.735899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:57.735744    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:58.013829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:58.013735    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:58.222705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:58.222673    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:58.349356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:58.349205    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:59.068391 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:59.068353    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:59.236985 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:59.236952    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:29:59.351767 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:29:59.351678    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:00.333366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.333336    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m"]
Oct 29 23:30:00.334325 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.334296    5428 cpu_manager.go:414] "containerMap: removing container" podUID="432a4527-a0fc-41dc-b131-4843d3d7e673" containerName="payload-processing"
Oct 29 23:30:00.334325 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.334315    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432a4527-a0fc-41dc-b131-4843d3d7e673" containerName="payload-processing"
Oct 29 23:30:00.334325 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.334334    5428 cpu_manager.go:414] "containerMap: removing container" podUID="5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" containerName="istio-proxy"
Oct 29 23:30:00.334603 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.334339    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" containerName="istio-proxy"
Oct 29 23:30:00.334603 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.334515    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="432a4527-a0fc-41dc-b131-4843d3d7e673" containerName="payload-processing"
Oct 29 23:30:00.334603 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.334526    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a372a69-eaeb-4f73-ba14-fb4fa70a9ee6" containerName="istio-proxy"
Oct 29 23:30:00.339132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.339107    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5"]
Oct 29 23:30:00.339284 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.339226    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m"
Oct 29 23:30:00.346210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.346179    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3860e17-218e-4d30-bc36-a4f23f66b3ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m\" (UID: \"c3860e17-218e-4d30-bc36-a4f23f66b3ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m"
Oct 29 23:30:00.346783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.346755    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vq2g\" (UniqueName: \"kubernetes.io/projected/c3860e17-218e-4d30-bc36-a4f23f66b3ed-kube-api-access-5vq2g\") pod \"maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m\" (UID: \"c3860e17-218e-4d30-bc36-a4f23f66b3ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m"
Oct 29 23:30:00.347168 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.346519    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5"
Oct 29 23:30:00.358970 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.357707    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m"]
Oct 29 23:30:00.361967 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.361923    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\""
Oct 29 23:30:00.384479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.384450    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5"]
Oct 29 23:30:00.392415 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.392385    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9"]
Oct 29 23:30:00.403217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.401881    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9"
Oct 29 23:30:00.407397 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.407370    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9"]
Oct 29 23:30:00.448702 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.448669    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3860e17-218e-4d30-bc36-a4f23f66b3ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m\" (UID: \"c3860e17-218e-4d30-bc36-a4f23f66b3ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m"
Oct 29 23:30:00.448702 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.448713    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw4dz\" (UniqueName: \"kubernetes.io/projected/4869263d-ba1c-40b0-9d05-aa7cbbeda452-kube-api-access-fw4dz\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5\" (UID: \"4869263d-ba1c-40b0-9d05-aa7cbbeda452\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5"
Oct 29 23:30:00.448956 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.448777    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vq2g\" (UniqueName: \"kubernetes.io/projected/c3860e17-218e-4d30-bc36-a4f23f66b3ed-kube-api-access-5vq2g\") pod \"maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m\" (UID: \"c3860e17-218e-4d30-bc36-a4f23f66b3ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m"
Oct 29 23:30:00.448956 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.448839    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnlc5\" (UniqueName: \"kubernetes.io/projected/3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3-kube-api-access-mnlc5\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9\" (UID: \"3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9"
Oct 29 23:30:00.448956 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.448942    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4869263d-ba1c-40b0-9d05-aa7cbbeda452-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5\" (UID: \"4869263d-ba1c-40b0-9d05-aa7cbbeda452\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5"
Oct 29 23:30:00.449106 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.448976    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9\" (UID: \"3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9"
Oct 29 23:30:00.449320 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.449300    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3860e17-218e-4d30-bc36-a4f23f66b3ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m\" (UID: \"c3860e17-218e-4d30-bc36-a4f23f66b3ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m"
Oct 29 23:30:00.471972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.471945    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vq2g\" (UniqueName: \"kubernetes.io/projected/c3860e17-218e-4d30-bc36-a4f23f66b3ed-kube-api-access-5vq2g\") pod \"maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m\" (UID: \"c3860e17-218e-4d30-bc36-a4f23f66b3ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m"
Oct 29 23:30:00.550319 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.550287    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnlc5\" (UniqueName: \"kubernetes.io/projected/3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3-kube-api-access-mnlc5\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9\" (UID: \"3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9"
Oct 29 23:30:00.550485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.550342    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4869263d-ba1c-40b0-9d05-aa7cbbeda452-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5\" (UID: \"4869263d-ba1c-40b0-9d05-aa7cbbeda452\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5"
Oct 29 23:30:00.550485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.550365    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9\" (UID: \"3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9"
Oct 29 23:30:00.550629 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.550550    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw4dz\" (UniqueName: \"kubernetes.io/projected/4869263d-ba1c-40b0-9d05-aa7cbbeda452-kube-api-access-fw4dz\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5\" (UID: \"4869263d-ba1c-40b0-9d05-aa7cbbeda452\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5"
Oct 29 23:30:00.551073 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.551047    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9\" (UID: \"3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9"
Oct 29 23:30:00.551176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.551053    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4869263d-ba1c-40b0-9d05-aa7cbbeda452-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5\" (UID: \"4869263d-ba1c-40b0-9d05-aa7cbbeda452\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5"
Oct 29 23:30:00.573424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.573396    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw4dz\" (UniqueName: \"kubernetes.io/projected/4869263d-ba1c-40b0-9d05-aa7cbbeda452-kube-api-access-fw4dz\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5\" (UID: \"4869263d-ba1c-40b0-9d05-aa7cbbeda452\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5"
Oct 29 23:30:00.573541 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.573404    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnlc5\" (UniqueName: \"kubernetes.io/projected/3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3-kube-api-access-mnlc5\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9\" (UID: \"3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9"
Oct 29 23:30:00.630097 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.630069    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:00.670213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.670180    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m"
Oct 29 23:30:00.679956 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.679929    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5"
Oct 29 23:30:00.720926 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:00.720896    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9"
Oct 29 23:30:01.020160 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:30:01.020129    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3860e17_218e_4d30_bc36_a4f23f66b3ed.slice/crio-8c63bff843bb68c1c505279f1f12184ae753fdcde350e7a16e22ee07a7fad26d WatchSource:0}: Error finding container 8c63bff843bb68c1c505279f1f12184ae753fdcde350e7a16e22ee07a7fad26d: Status 404 returned error can't find the container with id 8c63bff843bb68c1c505279f1f12184ae753fdcde350e7a16e22ee07a7fad26d
Oct 29 23:30:01.020551 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:01.020329    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m"]
Oct 29 23:30:01.033262 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:01.033229    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5"]
Oct 29 23:30:01.036217 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:30:01.036191    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4869263d_ba1c_40b0_9d05_aa7cbbeda452.slice/crio-ee0062b697a788e10fc81ef79217868f82b6f866ac1e67b4c924e2e51128dd89 WatchSource:0}: Error finding container ee0062b697a788e10fc81ef79217868f82b6f866ac1e67b4c924e2e51128dd89: Status 404 returned error can't find the container with id ee0062b697a788e10fc81ef79217868f82b6f866ac1e67b4c924e2e51128dd89
Oct 29 23:30:01.090541 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:30:01.090515    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a4a1bbc_0240_43a5_90ac_f3bc5beb2dd3.slice/crio-d19f232da22b31566b53b74f5b52bde23d6a4a4cbe35997e3cde64131b93b53b WatchSource:0}: Error finding container d19f232da22b31566b53b74f5b52bde23d6a4a4cbe35997e3cde64131b93b53b: Status 404 returned error can't find the container with id d19f232da22b31566b53b74f5b52bde23d6a4a4cbe35997e3cde64131b93b53b
Oct 29 23:30:01.091411 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:01.091388    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9"]
Oct 29 23:30:01.960993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:01.960955    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9" event={"ID":"3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3","Type":"ContainerStarted","Data":"2452822c8370554c89c8bb5cdb5c5e5fd57b11d3706f68914d3414cc33052d9f"}
Oct 29 23:30:01.961403 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:01.961030    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9" event={"ID":"3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3","Type":"ContainerStarted","Data":"d19f232da22b31566b53b74f5b52bde23d6a4a4cbe35997e3cde64131b93b53b"}
Oct 29 23:30:01.963397 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:01.963361    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m" event={"ID":"c3860e17-218e-4d30-bc36-a4f23f66b3ed","Type":"ContainerStarted","Data":"8a5d52f21fd052732a8c754730213d8fbadc960a51d48520287416641142cf46"}
Oct 29 23:30:01.963528 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:01.963425    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m" event={"ID":"c3860e17-218e-4d30-bc36-a4f23f66b3ed","Type":"ContainerStarted","Data":"8c63bff843bb68c1c505279f1f12184ae753fdcde350e7a16e22ee07a7fad26d"}
Oct 29 23:30:01.965296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:01.965273    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5" event={"ID":"4869263d-ba1c-40b0-9d05-aa7cbbeda452","Type":"ContainerStarted","Data":"bf1665b46cb9cecb8b662e7e4618d232682832d4874d072364472e7c5b5142df"}
Oct 29 23:30:01.965386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:01.965312    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5" event={"ID":"4869263d-ba1c-40b0-9d05-aa7cbbeda452","Type":"ContainerStarted","Data":"ee0062b697a788e10fc81ef79217868f82b6f866ac1e67b4c924e2e51128dd89"}
Oct 29 23:30:02.002650 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:02.002604    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9" podStartSLOduration=2.002590391 podStartE2EDuration="2.002590391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:02.001587407 +0000 UTC m=+2805.344536153" watchObservedRunningTime="2026-10-29 23:30:02.002590391 +0000 UTC m=+2805.345539136"
Oct 29 23:30:02.047888 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:02.047844    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5" podStartSLOduration=2.047827157 podStartE2EDuration="2.047827157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:02.044150247 +0000 UTC m=+2805.387098995" watchObservedRunningTime="2026-10-29 23:30:02.047827157 +0000 UTC m=+2805.390775908"
Oct 29 23:30:02.088495 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:02.086823    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m" podStartSLOduration=2.086790584 podStartE2EDuration="2.086790584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:02.082553631 +0000 UTC m=+2805.425502376" watchObservedRunningTime="2026-10-29 23:30:02.086790584 +0000 UTC m=+2805.429739330"
Oct 29 23:30:02.132763 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:02.132729    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:02.248820 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:02.248740    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:02.429599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:02.429563    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:02.979250 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:02.979213    5428 generic.go:356] "Generic (PLEG): container finished" podID="c3860e17-218e-4d30-bc36-a4f23f66b3ed" containerID="8a5d52f21fd052732a8c754730213d8fbadc960a51d48520287416641142cf46" exitCode=60
Oct 29 23:30:02.979833 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:02.979289    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m" event={"ID":"c3860e17-218e-4d30-bc36-a4f23f66b3ed","Type":"ContainerDied","Data":"8a5d52f21fd052732a8c754730213d8fbadc960a51d48520287416641142cf46"}
Oct 29 23:30:02.979833 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:02.979563    5428 scope.go:122] "RemoveContainer" containerID="8a5d52f21fd052732a8c754730213d8fbadc960a51d48520287416641142cf46"
Oct 29 23:30:02.981499 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:02.981469    5428 generic.go:356] "Generic (PLEG): container finished" podID="4869263d-ba1c-40b0-9d05-aa7cbbeda452" containerID="bf1665b46cb9cecb8b662e7e4618d232682832d4874d072364472e7c5b5142df" exitCode=60
Oct 29 23:30:02.981626 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:02.981576    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5" event={"ID":"4869263d-ba1c-40b0-9d05-aa7cbbeda452","Type":"ContainerDied","Data":"bf1665b46cb9cecb8b662e7e4618d232682832d4874d072364472e7c5b5142df"}
Oct 29 23:30:02.981835 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:02.981823    5428 scope.go:122] "RemoveContainer" containerID="bf1665b46cb9cecb8b662e7e4618d232682832d4874d072364472e7c5b5142df"
Oct 29 23:30:02.983632 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:02.983593    5428 generic.go:356] "Generic (PLEG): container finished" podID="3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3" containerID="2452822c8370554c89c8bb5cdb5c5e5fd57b11d3706f68914d3414cc33052d9f" exitCode=60
Oct 29 23:30:02.984184 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:02.983939    5428 scope.go:122] "RemoveContainer" containerID="2452822c8370554c89c8bb5cdb5c5e5fd57b11d3706f68914d3414cc33052d9f"
Oct 29 23:30:02.984526 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:02.984494    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9" event={"ID":"3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3","Type":"ContainerDied","Data":"2452822c8370554c89c8bb5cdb5c5e5fd57b11d3706f68914d3414cc33052d9f"}
Oct 29 23:30:03.355998 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:03.355899    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:03.603871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:03.603775    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:03.737663 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:03.737631    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:04.013935 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:04.013835    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m" event={"ID":"c3860e17-218e-4d30-bc36-a4f23f66b3ed","Type":"ContainerStarted","Data":"160f2e3ec61c2d41c38a924a6576b5d31314b4d1bf96185a5c5dae7f13de34b0"}
Oct 29 23:30:04.016961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:04.016910    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5" event={"ID":"4869263d-ba1c-40b0-9d05-aa7cbbeda452","Type":"ContainerStarted","Data":"d58493c607a0d069ed797601d70ecec41dda3ee15b9789fe67915989df9633f6"}
Oct 29 23:30:04.019443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:04.019412    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9" event={"ID":"3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3","Type":"ContainerStarted","Data":"932d5bf6fa005d6529406912ddf4e07162234bf951a911af453edbd692808a1f"}
Oct 29 23:30:04.580475 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:04.580441    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:04.701535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:04.701507    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:04.906217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:04.906187    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:05.040822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:05.039845    5428 generic.go:356] "Generic (PLEG): container finished" podID="c3860e17-218e-4d30-bc36-a4f23f66b3ed" containerID="160f2e3ec61c2d41c38a924a6576b5d31314b4d1bf96185a5c5dae7f13de34b0" exitCode=60
Oct 29 23:30:05.040822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:05.040044    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m" event={"ID":"c3860e17-218e-4d30-bc36-a4f23f66b3ed","Type":"ContainerDied","Data":"160f2e3ec61c2d41c38a924a6576b5d31314b4d1bf96185a5c5dae7f13de34b0"}
Oct 29 23:30:05.040822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:05.040102    5428 scope.go:122] "RemoveContainer" containerID="8a5d52f21fd052732a8c754730213d8fbadc960a51d48520287416641142cf46"
Oct 29 23:30:05.043846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:05.041777    5428 scope.go:122] "RemoveContainer" containerID="160f2e3ec61c2d41c38a924a6576b5d31314b4d1bf96185a5c5dae7f13de34b0"
Oct 29 23:30:05.043846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:05.043448    5428 generic.go:356] "Generic (PLEG): container finished" podID="4869263d-ba1c-40b0-9d05-aa7cbbeda452" containerID="d58493c607a0d069ed797601d70ecec41dda3ee15b9789fe67915989df9633f6" exitCode=60
Oct 29 23:30:05.043846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:05.043538    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5" event={"ID":"4869263d-ba1c-40b0-9d05-aa7cbbeda452","Type":"ContainerDied","Data":"d58493c607a0d069ed797601d70ecec41dda3ee15b9789fe67915989df9633f6"}
Oct 29 23:30:05.044429 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:30:05.044255    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m_odh-ai-gateway-infra(c3860e17-218e-4d30-bc36-a4f23f66b3ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m" podUID="c3860e17-218e-4d30-bc36-a4f23f66b3ed"
Oct 29 23:30:05.044682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:05.044664    5428 scope.go:122] "RemoveContainer" containerID="d58493c607a0d069ed797601d70ecec41dda3ee15b9789fe67915989df9633f6"
Oct 29 23:30:05.045082 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:30:05.045058    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5_odh-ai-gateway-infra(4869263d-ba1c-40b0-9d05-aa7cbbeda452)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5" podUID="4869263d-ba1c-40b0-9d05-aa7cbbeda452"
Oct 29 23:30:05.046479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:05.046453    5428 generic.go:356] "Generic (PLEG): container finished" podID="3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3" containerID="932d5bf6fa005d6529406912ddf4e07162234bf951a911af453edbd692808a1f" exitCode=60
Oct 29 23:30:05.046637 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:05.046621    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9" event={"ID":"3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3","Type":"ContainerDied","Data":"932d5bf6fa005d6529406912ddf4e07162234bf951a911af453edbd692808a1f"}
Oct 29 23:30:05.047071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:05.047048    5428 scope.go:122] "RemoveContainer" containerID="932d5bf6fa005d6529406912ddf4e07162234bf951a911af453edbd692808a1f"
Oct 29 23:30:05.047426 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:30:05.047400    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9_odh-ai-gateway-infra(3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9" podUID="3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3"
Oct 29 23:30:05.063641 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:05.063618    5428 scope.go:122] "RemoveContainer" containerID="bf1665b46cb9cecb8b662e7e4618d232682832d4874d072364472e7c5b5142df"
Oct 29 23:30:05.078718 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:05.078695    5428 scope.go:122] "RemoveContainer" containerID="2452822c8370554c89c8bb5cdb5c5e5fd57b11d3706f68914d3414cc33052d9f"
Oct 29 23:30:05.546330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:05.546293    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:05.669833 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:05.669781    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:06.060125 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:06.060095    5428 scope.go:122] "RemoveContainer" containerID="160f2e3ec61c2d41c38a924a6576b5d31314b4d1bf96185a5c5dae7f13de34b0"
Oct 29 23:30:06.060629 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:30:06.060495    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m_odh-ai-gateway-infra(c3860e17-218e-4d30-bc36-a4f23f66b3ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m" podUID="c3860e17-218e-4d30-bc36-a4f23f66b3ed"
Oct 29 23:30:06.068518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:06.068493    5428 scope.go:122] "RemoveContainer" containerID="d58493c607a0d069ed797601d70ecec41dda3ee15b9789fe67915989df9633f6"
Oct 29 23:30:06.068995 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:30:06.068788    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5_odh-ai-gateway-infra(4869263d-ba1c-40b0-9d05-aa7cbbeda452)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5" podUID="4869263d-ba1c-40b0-9d05-aa7cbbeda452"
Oct 29 23:30:06.070846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:06.070822    5428 scope.go:122] "RemoveContainer" containerID="932d5bf6fa005d6529406912ddf4e07162234bf951a911af453edbd692808a1f"
Oct 29 23:30:06.071132 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:30:06.071109    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9_odh-ai-gateway-infra(3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9" podUID="3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3"
Oct 29 23:30:07.191953 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:07.191922    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:07.666535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:07.666501    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:07.859403 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:07.859364    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:08.165951 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:08.165909    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:08.324920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:08.324882    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:08.450101 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:08.450019    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:08.602950 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:08.602908    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:08.723841 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:08.723758    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:09.114138 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:09.114112    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:09.607098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:09.607010    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:11.368075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:11.368040    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:13.084722 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:13.084692    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:14.769637 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:14.769601    5428 scope.go:122] "RemoveContainer" containerID="d58493c607a0d069ed797601d70ecec41dda3ee15b9789fe67915989df9633f6"
Oct 29 23:30:14.770141 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:14.769754    5428 scope.go:122] "RemoveContainer" containerID="932d5bf6fa005d6529406912ddf4e07162234bf951a911af453edbd692808a1f"
Oct 29 23:30:14.825690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:14.825662    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:14.940854 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:14.940820    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:15.183945 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:15.183917    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5" event={"ID":"4869263d-ba1c-40b0-9d05-aa7cbbeda452","Type":"ContainerStarted","Data":"3b84239fd4310919bef1fde22166a24687917e5f2ca8c95d99a9551ea28c455f"}
Oct 29 23:30:15.185869 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:15.185846    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9" event={"ID":"3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3","Type":"ContainerStarted","Data":"348dc0c4f3e2666723d9c32255008171c74f917fe40142d2957893b55c2ab610"}
Oct 29 23:30:15.770013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:15.769960    5428 scope.go:122] "RemoveContainer" containerID="160f2e3ec61c2d41c38a924a6576b5d31314b4d1bf96185a5c5dae7f13de34b0"
Oct 29 23:30:15.774295 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:15.773677    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:16.005454 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:16.005407    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:16.196771 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:16.196734    5428 generic.go:356] "Generic (PLEG): container finished" podID="4869263d-ba1c-40b0-9d05-aa7cbbeda452" containerID="3b84239fd4310919bef1fde22166a24687917e5f2ca8c95d99a9551ea28c455f" exitCode=60
Oct 29 23:30:16.196984 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:16.196780    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5" event={"ID":"4869263d-ba1c-40b0-9d05-aa7cbbeda452","Type":"ContainerDied","Data":"3b84239fd4310919bef1fde22166a24687917e5f2ca8c95d99a9551ea28c455f"}
Oct 29 23:30:16.196984 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:16.196854    5428 scope.go:122] "RemoveContainer" containerID="d58493c607a0d069ed797601d70ecec41dda3ee15b9789fe67915989df9633f6"
Oct 29 23:30:16.197318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:16.197300    5428 scope.go:122] "RemoveContainer" containerID="3b84239fd4310919bef1fde22166a24687917e5f2ca8c95d99a9551ea28c455f"
Oct 29 23:30:16.197740 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:30:16.197719    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5_odh-ai-gateway-infra(4869263d-ba1c-40b0-9d05-aa7cbbeda452)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5" podUID="4869263d-ba1c-40b0-9d05-aa7cbbeda452"
Oct 29 23:30:16.199066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:16.199045    5428 generic.go:356] "Generic (PLEG): container finished" podID="3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3" containerID="348dc0c4f3e2666723d9c32255008171c74f917fe40142d2957893b55c2ab610" exitCode=60
Oct 29 23:30:16.199150 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:16.199090    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9" event={"ID":"3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3","Type":"ContainerDied","Data":"348dc0c4f3e2666723d9c32255008171c74f917fe40142d2957893b55c2ab610"}
Oct 29 23:30:16.199529 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:16.199507    5428 scope.go:122] "RemoveContainer" containerID="348dc0c4f3e2666723d9c32255008171c74f917fe40142d2957893b55c2ab610"
Oct 29 23:30:16.199910 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:30:16.199890    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9_odh-ai-gateway-infra(3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9" podUID="3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3"
Oct 29 23:30:16.212906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:16.212886    5428 scope.go:122] "RemoveContainer" containerID="932d5bf6fa005d6529406912ddf4e07162234bf951a911af453edbd692808a1f"
Oct 29 23:30:16.267692 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:16.267658    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5"]
Oct 29 23:30:16.352926 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:16.352886    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9"]
Oct 29 23:30:17.215912 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.215874    5428 generic.go:356] "Generic (PLEG): container finished" podID="c3860e17-218e-4d30-bc36-a4f23f66b3ed" containerID="e224b99bd8e36b24e8cde804db0956df59bcfaea5b7d0a47507c3aed055db518" exitCode=60
Oct 29 23:30:17.216345 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.215954    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m" event={"ID":"c3860e17-218e-4d30-bc36-a4f23f66b3ed","Type":"ContainerDied","Data":"e224b99bd8e36b24e8cde804db0956df59bcfaea5b7d0a47507c3aed055db518"}
Oct 29 23:30:17.216345 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.216009    5428 scope.go:122] "RemoveContainer" containerID="160f2e3ec61c2d41c38a924a6576b5d31314b4d1bf96185a5c5dae7f13de34b0"
Oct 29 23:30:17.217024 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.216998    5428 scope.go:122] "RemoveContainer" containerID="e224b99bd8e36b24e8cde804db0956df59bcfaea5b7d0a47507c3aed055db518"
Oct 29 23:30:17.217575 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:30:17.217524    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m_odh-ai-gateway-infra(c3860e17-218e-4d30-bc36-a4f23f66b3ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m" podUID="c3860e17-218e-4d30-bc36-a4f23f66b3ed"
Oct 29 23:30:17.299845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.299619    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:17.415449 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:30:17.415370    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:30:17.497309 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.496919    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9"
Oct 29 23:30:17.535878 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.535852    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5"
Oct 29 23:30:17.566983 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.566953    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3-maas-api-service-ca\") pod \"3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3\" (UID: \"3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3\") "
Oct 29 23:30:17.567221 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.567047    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4869263d-ba1c-40b0-9d05-aa7cbbeda452-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4869263d-ba1c-40b0-9d05-aa7cbbeda452-maas-api-service-ca\") pod \"4869263d-ba1c-40b0-9d05-aa7cbbeda452\" (UID: \"4869263d-ba1c-40b0-9d05-aa7cbbeda452\") "
Oct 29 23:30:17.567366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.567344    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4869263d-ba1c-40b0-9d05-aa7cbbeda452-kube-api-access-fw4dz\" (UniqueName: \"kubernetes.io/projected/4869263d-ba1c-40b0-9d05-aa7cbbeda452-kube-api-access-fw4dz\") pod \"4869263d-ba1c-40b0-9d05-aa7cbbeda452\" (UID: \"4869263d-ba1c-40b0-9d05-aa7cbbeda452\") "
Oct 29 23:30:17.567445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.567384    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3-kube-api-access-mnlc5\" (UniqueName: \"kubernetes.io/projected/3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3-kube-api-access-mnlc5\") pod \"3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3\" (UID: \"3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3\") "
Oct 29 23:30:17.567896 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.567796    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3-maas-api-service-ca" pod "3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3" (UID: "3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:30:17.568205 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.568182    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:30:17.568725 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.568703    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4869263d-ba1c-40b0-9d05-aa7cbbeda452-maas-api-service-ca" pod "4869263d-ba1c-40b0-9d05-aa7cbbeda452" (UID: "4869263d-ba1c-40b0-9d05-aa7cbbeda452"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:30:17.570753 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.570724    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3-kube-api-access-mnlc5" pod "3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3" (UID: "3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3"). InnerVolumeSpecName "kube-api-access-mnlc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:30:17.571056 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.571037    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4869263d-ba1c-40b0-9d05-aa7cbbeda452-kube-api-access-fw4dz" pod "4869263d-ba1c-40b0-9d05-aa7cbbeda452" (UID: "4869263d-ba1c-40b0-9d05-aa7cbbeda452"). InnerVolumeSpecName "kube-api-access-fw4dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:30:17.669370 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.669330    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw4dz\" (UniqueName: \"kubernetes.io/projected/4869263d-ba1c-40b0-9d05-aa7cbbeda452-kube-api-access-fw4dz\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:30:17.669370 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.669357    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnlc5\" (UniqueName: \"kubernetes.io/projected/3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3-kube-api-access-mnlc5\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:30:17.669370 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:17.669367    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4869263d-ba1c-40b0-9d05-aa7cbbeda452-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:30:18.227506 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.227426    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5"
Oct 29 23:30:18.227506 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.227444    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5" event={"ID":"4869263d-ba1c-40b0-9d05-aa7cbbeda452","Type":"ContainerDied","Data":"ee0062b697a788e10fc81ef79217868f82b6f866ac1e67b4c924e2e51128dd89"}
Oct 29 23:30:18.228006 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.227509    5428 scope.go:122] "RemoveContainer" containerID="3b84239fd4310919bef1fde22166a24687917e5f2ca8c95d99a9551ea28c455f"
Oct 29 23:30:18.229284 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.229261    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9"
Oct 29 23:30:18.229434 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.229280    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9" event={"ID":"3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3","Type":"ContainerDied","Data":"d19f232da22b31566b53b74f5b52bde23d6a4a4cbe35997e3cde64131b93b53b"}
Oct 29 23:30:18.241232 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.241215    5428 scope.go:122] "RemoveContainer" containerID="348dc0c4f3e2666723d9c32255008171c74f917fe40142d2957893b55c2ab610"
Oct 29 23:30:18.281398 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.281371    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9"]
Oct 29 23:30:18.293297 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.293272    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10975-29888610-jjgr9"]
Oct 29 23:30:18.314558 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.314531    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m"]
Oct 29 23:30:18.327833 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.327578    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5"]
Oct 29 23:30:18.337627 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.337594    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888610-l9cc5"]
Oct 29 23:30:18.494089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.494022    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m"
Oct 29 23:30:18.581764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.581722    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3860e17-218e-4d30-bc36-a4f23f66b3ed-kube-api-access-5vq2g\" (UniqueName: \"kubernetes.io/projected/c3860e17-218e-4d30-bc36-a4f23f66b3ed-kube-api-access-5vq2g\") pod \"c3860e17-218e-4d30-bc36-a4f23f66b3ed\" (UID: \"c3860e17-218e-4d30-bc36-a4f23f66b3ed\") "
Oct 29 23:30:18.582118 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.582093    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3860e17-218e-4d30-bc36-a4f23f66b3ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3860e17-218e-4d30-bc36-a4f23f66b3ed-maas-api-service-ca\") pod \"c3860e17-218e-4d30-bc36-a4f23f66b3ed\" (UID: \"c3860e17-218e-4d30-bc36-a4f23f66b3ed\") "
Oct 29 23:30:18.582550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.582524    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3860e17-218e-4d30-bc36-a4f23f66b3ed-maas-api-service-ca" pod "c3860e17-218e-4d30-bc36-a4f23f66b3ed" (UID: "c3860e17-218e-4d30-bc36-a4f23f66b3ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:30:18.582817 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.582782    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3860e17-218e-4d30-bc36-a4f23f66b3ed-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:30:18.584022 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.583994    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3860e17-218e-4d30-bc36-a4f23f66b3ed-kube-api-access-5vq2g" pod "c3860e17-218e-4d30-bc36-a4f23f66b3ed" (UID: "c3860e17-218e-4d30-bc36-a4f23f66b3ed"). InnerVolumeSpecName "kube-api-access-5vq2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:30:18.684649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.684621    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vq2g\" (UniqueName: \"kubernetes.io/projected/c3860e17-218e-4d30-bc36-a4f23f66b3ed-kube-api-access-5vq2g\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:30:18.779819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.779733    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3" path="/var/lib/kubelet/pods/3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3/volumes"
Oct 29 23:30:18.780297 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:18.780279    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4869263d-ba1c-40b0-9d05-aa7cbbeda452" path="/var/lib/kubelet/pods/4869263d-ba1c-40b0-9d05-aa7cbbeda452/volumes"
Oct 29 23:30:19.186464 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:19.186433    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:19.244493 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:19.244416    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m"
Oct 29 23:30:19.244493 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:19.244471    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m" event={"ID":"c3860e17-218e-4d30-bc36-a4f23f66b3ed","Type":"ContainerDied","Data":"8c63bff843bb68c1c505279f1f12184ae753fdcde350e7a16e22ee07a7fad26d"}
Oct 29 23:30:19.245266 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:19.244554    5428 scope.go:122] "RemoveContainer" containerID="e224b99bd8e36b24e8cde804db0956df59bcfaea5b7d0a47507c3aed055db518"
Oct 29 23:30:19.306119 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:19.306083    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m"]
Oct 29 23:30:19.318223 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:19.318196    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c5ef2f-29888610-5745m"]
Oct 29 23:30:20.141376 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:20.141337    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:20.279389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:20.279359    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:20.780790 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:20.780749    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3860e17-218e-4d30-bc36-a4f23f66b3ed" path="/var/lib/kubelet/pods/c3860e17-218e-4d30-bc36-a4f23f66b3ed/volumes"
Oct 29 23:30:22.292222 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:22.292175    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:22.848190 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:22.848153    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:23.008620 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:23.008585    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:23.168969 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:23.168936    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:23.313264 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:23.313227    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:26.155041 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:26.155009    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:26.280118 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:26.280081    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:26.681736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:26.681696    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:26.811993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:26.811955    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:28.088972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:28.088935    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:30.213928 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:30.213896    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:30.792886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:30.792851    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:30.917908 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:30.917849    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:31.058155 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:31.058069    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:31.680767 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:31.680734    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:32.210177 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:32.210145    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:32.318905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:32.318875    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:32.441602 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:32.441575    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:32.564040 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:32.563949    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:34.357949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:34.357909    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:36.109883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:36.109849    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:37.915951 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:37.915914    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:39.778933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:39.778895    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:41.565143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:41.565112    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:43.426310 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:43.426279    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:45.653571 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:45.653535    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:47.381325 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:30:47.381283    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:30:48.103887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:48.103839    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:49.304698 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:49.304601    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:52.197066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:52.197029    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:54.399955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:54.399925    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:56.197732 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:56.197701    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:58.090591 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:58.090559    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:58.208311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:58.208276    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:59.666946 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:59.666912    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:30:59.865247 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:30:59.864938    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:00.501469 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:00.501435    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:00.695653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:00.695618    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:03.016796 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:03.016760    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:04.843783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:04.843746    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:05.187529 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:05.187499    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:05.308081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:05.308045    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:07.271503 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:07.271473    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:09.054550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:09.054518    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:10.801253 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:10.801217    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:12.544734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:12.544703    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:14.300615 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:14.300578    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:16.090212 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:16.090170    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:17.141674 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:17.141640    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:17.370980 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:31:17.370944    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:31:17.614662 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:17.614622    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:17.975682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:17.975605    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:18.320622 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:18.320525    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:19.845356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:19.845328    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:21.112270 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:21.112239    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:21.663349 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:21.663313    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:22.046201 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:22.045796    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:22.180591 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:22.180556    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:23.688506 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:23.688464    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:23.864043 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:23.864009    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:25.719220 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:25.719190    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:27.481176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:27.481142    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:29.378738 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:29.378706    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:31.132298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:31.132263    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:31.247345 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:31.247314    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:32.969544 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:32.969505    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:34.792587 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:34.792554    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:36.517928 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:36.517895    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:38.264779 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.264744    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"]
Oct 29 23:31:38.265681 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265663    5428 cpu_manager.go:414] "containerMap: removing container" podUID="4869263d-ba1c-40b0-9d05-aa7cbbeda452" containerName="cleanup"
Oct 29 23:31:38.265681 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265680    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4869263d-ba1c-40b0-9d05-aa7cbbeda452" containerName="cleanup"
Oct 29 23:31:38.265793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265692    5428 cpu_manager.go:414] "containerMap: removing container" podUID="c3860e17-218e-4d30-bc36-a4f23f66b3ed" containerName="cleanup"
Oct 29 23:31:38.265793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265698    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3860e17-218e-4d30-bc36-a4f23f66b3ed" containerName="cleanup"
Oct 29 23:31:38.265793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265711    5428 cpu_manager.go:414] "containerMap: removing container" podUID="4869263d-ba1c-40b0-9d05-aa7cbbeda452" containerName="cleanup"
Oct 29 23:31:38.265793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265716    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4869263d-ba1c-40b0-9d05-aa7cbbeda452" containerName="cleanup"
Oct 29 23:31:38.265793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265724    5428 cpu_manager.go:414] "containerMap: removing container" podUID="c3860e17-218e-4d30-bc36-a4f23f66b3ed" containerName="cleanup"
Oct 29 23:31:38.265793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265729    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3860e17-218e-4d30-bc36-a4f23f66b3ed" containerName="cleanup"
Oct 29 23:31:38.265793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265763    5428 cpu_manager.go:414] "containerMap: removing container" podUID="4869263d-ba1c-40b0-9d05-aa7cbbeda452" containerName="cleanup"
Oct 29 23:31:38.265793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265767    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4869263d-ba1c-40b0-9d05-aa7cbbeda452" containerName="cleanup"
Oct 29 23:31:38.265793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265778    5428 cpu_manager.go:414] "containerMap: removing container" podUID="3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3" containerName="cleanup"
Oct 29 23:31:38.265793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265783    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3" containerName="cleanup"
Oct 29 23:31:38.265793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265796    5428 cpu_manager.go:414] "containerMap: removing container" podUID="3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3" containerName="cleanup"
Oct 29 23:31:38.265793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265815    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3" containerName="cleanup"
Oct 29 23:31:38.265793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265826    5428 cpu_manager.go:414] "containerMap: removing container" podUID="3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3" containerName="cleanup"
Oct 29 23:31:38.266313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265833    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3" containerName="cleanup"
Oct 29 23:31:38.266313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265970    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4869263d-ba1c-40b0-9d05-aa7cbbeda452" containerName="cleanup"
Oct 29 23:31:38.266313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265978    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3860e17-218e-4d30-bc36-a4f23f66b3ed" containerName="cleanup"
Oct 29 23:31:38.266313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265986    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4869263d-ba1c-40b0-9d05-aa7cbbeda452" containerName="cleanup"
Oct 29 23:31:38.266313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.265993    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3860e17-218e-4d30-bc36-a4f23f66b3ed" containerName="cleanup"
Oct 29 23:31:38.266313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.266005    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3860e17-218e-4d30-bc36-a4f23f66b3ed" containerName="cleanup"
Oct 29 23:31:38.266313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.266013    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3" containerName="cleanup"
Oct 29 23:31:38.266313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.266021    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3" containerName="cleanup"
Oct 29 23:31:38.266313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.266026    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a4a1bbc-0240-43a5-90ac-f3bc5beb2dd3" containerName="cleanup"
Oct 29 23:31:38.270201 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.270182    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"
Oct 29 23:31:38.291327 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.291294    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"]
Oct 29 23:31:38.296626 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.296595    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-90f683-serving-cert\""
Oct 29 23:31:38.357461 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.357420    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:38.450511 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.450474    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-openshift-service-ca\") pod \"maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq\" (UID: \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"
Oct 29 23:31:38.450970 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.450938    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-metrics-certs\") pod \"maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq\" (UID: \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"
Oct 29 23:31:38.451370 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.451345    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhclz\" (UniqueName: \"kubernetes.io/projected/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-kube-api-access-nhclz\") pod \"maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq\" (UID: \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"
Oct 29 23:31:38.451564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.451543    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-maas-api-tls\") pod \"maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq\" (UID: \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"
Oct 29 23:31:38.469458 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.468078    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:38.553177 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.553083    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-metrics-certs\") pod \"maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq\" (UID: \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"
Oct 29 23:31:38.553422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.553252    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhclz\" (UniqueName: \"kubernetes.io/projected/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-kube-api-access-nhclz\") pod \"maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq\" (UID: \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"
Oct 29 23:31:38.553422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.553275    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-maas-api-tls\") pod \"maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq\" (UID: \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"
Oct 29 23:31:38.553422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.553359    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-openshift-service-ca\") pod \"maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq\" (UID: \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"
Oct 29 23:31:38.554105 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.554082    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-openshift-service-ca\") pod \"maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq\" (UID: \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"
Oct 29 23:31:38.555883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.555857    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-metrics-certs\") pod \"maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq\" (UID: \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"
Oct 29 23:31:38.556049 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.556031    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-maas-api-tls\") pod \"maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq\" (UID: \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"
Oct 29 23:31:38.577222 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.577187    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhclz\" (UniqueName: \"kubernetes.io/projected/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-kube-api-access-nhclz\") pod \"maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq\" (UID: \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"
Oct 29 23:31:38.585166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.585138    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"
Oct 29 23:31:38.603741 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.603710    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:38.902273 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.902241    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"]
Oct 29 23:31:38.902917 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:31:38.902510    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6420f5b_f272_44f6_86c9_f0f87cafe7ff.slice/crio-d2c2e70d31be5fe16f3a8b9cf80a70a36f6a7b85237aca0d6e59c9211edbab21 WatchSource:0}: Error finding container d2c2e70d31be5fe16f3a8b9cf80a70a36f6a7b85237aca0d6e59c9211edbab21: Status 404 returned error can't find the container with id d2c2e70d31be5fe16f3a8b9cf80a70a36f6a7b85237aca0d6e59c9211edbab21
Oct 29 23:31:38.903995 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:38.903975    5428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:31:39.090061 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:39.090027    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq" event={"ID":"f6420f5b-f272-44f6-86c9-f0f87cafe7ff","Type":"ContainerStarted","Data":"d2c2e70d31be5fe16f3a8b9cf80a70a36f6a7b85237aca0d6e59c9211edbab21"}
Oct 29 23:31:40.099843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:40.099777    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq" event={"ID":"f6420f5b-f272-44f6-86c9-f0f87cafe7ff","Type":"ContainerStarted","Data":"a633be99f4b470aa875e562bb4aff8cd5d71ff167154d3222d29df83b906dd2b"}
Oct 29 23:31:40.100323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:40.100011    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"
Oct 29 23:31:40.144353 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:40.144306    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq" podStartSLOduration=1.70250229 podStartE2EDuration="2.144293262s" totalImagesPullingTime="441.790972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:40.140888766 +0000 UTC m=+2903.483837512" watchObservedRunningTime="2026-10-29 23:31:40.144293262 +0000 UTC m=+2903.487242007"
Oct 29 23:31:40.287275 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:40.287225    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:42.108719 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:42.108682    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:43.899781 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:43.899732    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:46.116406 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:46.116366    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"
Oct 29 23:31:47.445304 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:31:47.445229    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:31:47.724880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:47.724206    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:50.762677 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:50.762629    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:52.816173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:52.816125    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:54.486947 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:54.486913    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:56.203080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:56.203041    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:57.978301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:57.978264    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:31:59.857369 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:31:59.857334    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:00.733983 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:00.733943    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:00.857947 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:00.857917    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:02.225463 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:02.225426    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:02.822283 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:02.822250    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:03.308826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:03.308753    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:04.054991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:04.054904    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:04.305597 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:04.305506    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:06.379661 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:06.379626    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:07.113649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:07.113619    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:07.246035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:07.246003    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:08.299369 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:08.299338    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:08.781591 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:08.781553    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:08.901596 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:08.901533    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:10.462509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:10.462476    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:11.643411 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:11.643376    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:11.760048 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:11.760003    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:13.096787 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:13.096743    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:14.958820 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:14.958769    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:16.663858 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:16.663826    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:17.369922 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:32:17.369882    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:32:18.399998 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:18.399967    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:20.159509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:20.159460    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:21.882313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:21.882280    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:23.726991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:23.726956    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:23.852610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:23.852578    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:26.247088 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:26.247055    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:28.129780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:28.129743    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:30.066070 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:30.066036    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:31.258567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:31.258537    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:31.741301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:31.741269    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:31.868980 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:31.868949    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:31.987444 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:31.987415    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:32.193289 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:32.193255    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:33.694596 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:33.694562    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:35.565517 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:35.565481    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:36.539029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:36.538994    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:36.737609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:36.737577    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:37.117581 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:37.117498    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:37.991745 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:37.991710    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:39.782007 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:39.781966    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:41.580072 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:41.580037    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:43.520987 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:43.520954    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:43.653731 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:43.653701    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:43.771854 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:43.771748    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:46.442963 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:46.442899    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:47.368999 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:32:47.368960    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:32:48.515047 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:48.515014    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:49.990658 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:49.990621    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:50.136526 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:50.136482    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:50.265813 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:50.265730    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:50.396754 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:50.396724    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:50.522544 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:50.522454    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:50.823537 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:50.823444    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:51.442551 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:51.442520    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:51.575242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:51.575208    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:53.282869 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:53.282835    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:53.398162 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:53.398125    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:54.115137 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:54.115090    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:56.154633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:56.154595    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:56.263544 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:56.263506    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:56.548981 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:56.548896    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:57.119368 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.119331    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"]
Oct 29 23:32:57.120418 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.120398    5428 cpu_manager.go:414] "containerMap: removing container" podUID="c3860e17-218e-4d30-bc36-a4f23f66b3ed" containerName="cleanup"
Oct 29 23:32:57.120418 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.120419    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3860e17-218e-4d30-bc36-a4f23f66b3ed" containerName="cleanup"
Oct 29 23:32:57.120570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.120560    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4869263d-ba1c-40b0-9d05-aa7cbbeda452" containerName="cleanup"
Oct 29 23:32:57.124950 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.124930    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.142392 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.142371    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-90f683-openshift-default-dockercfg-tjqwz\""
Oct 29 23:32:57.160113 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.160088    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"]
Oct 29 23:32:57.234465 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.234435    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db63574f-0719-4902-989b-aac4b2d5c994-istio-token\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.234630 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.234497    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-istio-envoy\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.234630 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.234616    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hr42\" (UniqueName: \"kubernetes.io/projected/db63574f-0719-4902-989b-aac4b2d5c994-kube-api-access-7hr42\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.234740 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.234721    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-credential-socket\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.234810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.234787    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-workload-socket\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.234874 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.234826    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db63574f-0719-4902-989b-aac4b2d5c994-istio-podinfo\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.234920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.234908    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db63574f-0719-4902-989b-aac4b2d5c994-istiod-ca-cert\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.234966 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.234951    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-workload-certs\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.235037 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.235024    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-istio-data\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.337029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.336995    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-credential-socket\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.337239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.337066    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-workload-socket\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.337239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.337092    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db63574f-0719-4902-989b-aac4b2d5c994-istio-podinfo\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.337239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.337127    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db63574f-0719-4902-989b-aac4b2d5c994-istiod-ca-cert\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.337239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.337176    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-workload-certs\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.337480 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.337255    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-istio-data\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.337480 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.337367    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db63574f-0719-4902-989b-aac4b2d5c994-istio-token\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.337480 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.337450    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-istio-envoy\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.337649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.337476    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-credential-socket\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.337649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.337518    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-workload-socket\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.337649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.337580    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hr42\" (UniqueName: \"kubernetes.io/projected/db63574f-0719-4902-989b-aac4b2d5c994-kube-api-access-7hr42\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.337893 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.337847    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db63574f-0719-4902-989b-aac4b2d5c994-istiod-ca-cert\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.338026 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.337949    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-workload-certs\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.338026 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.337992    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-istio-data\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.339631 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.339610    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-istio-envoy\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.339746 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.339700    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db63574f-0719-4902-989b-aac4b2d5c994-istio-podinfo\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.360637 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.360613    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hr42\" (UniqueName: \"kubernetes.io/projected/db63574f-0719-4902-989b-aac4b2d5c994-kube-api-access-7hr42\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.361071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.361052    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db63574f-0719-4902-989b-aac4b2d5c994-istio-token\") pod \"e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") " pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.440327 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.440251    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:57.772823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.772781    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"]
Oct 29 23:32:57.777381 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:32:57.777348    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb63574f_0719_4902_989b_aac4b2d5c994.slice/crio-a45f6aaf21e168db759728aec70429d1f0ed1dda54ca2f2ac2ccd5fea79ef624 WatchSource:0}: Error finding container a45f6aaf21e168db759728aec70429d1f0ed1dda54ca2f2ac2ccd5fea79ef624: Status 404 returned error can't find the container with id a45f6aaf21e168db759728aec70429d1f0ed1dda54ca2f2ac2ccd5fea79ef624
Oct 29 23:32:57.781396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.781361    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:32:57.781524 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.781431    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:32:57.781524 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:57.781464    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:32:58.024590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:58.024503    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t" event={"ID":"db63574f-0719-4902-989b-aac4b2d5c994","Type":"ContainerStarted","Data":"fbbb2f6ec9d920466cdd77330c38f3b41a044a2f56712d838538cb014aabaa77"}
Oct 29 23:32:58.024590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:58.024567    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t" event={"ID":"db63574f-0719-4902-989b-aac4b2d5c994","Type":"ContainerStarted","Data":"a45f6aaf21e168db759728aec70429d1f0ed1dda54ca2f2ac2ccd5fea79ef624"}
Oct 29 23:32:58.072465 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:58.072401    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t" podStartSLOduration=1.072383116 podStartE2EDuration="1.072383116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:58.069546836 +0000 UTC m=+2981.412495582" watchObservedRunningTime="2026-10-29 23:32:58.072383116 +0000 UTC m=+2981.415331866"
Oct 29 23:32:58.440940 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:58.440900    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:32:58.445149 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:58.444776    5428 patch_prober.go:29] interesting pod/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 29 23:32:58.445149 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:58.444855    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t" podUID="db63574f-0719-4902-989b-aac4b2d5c994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 29 23:32:58.504031 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:58.503996    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:32:59.440618 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:59.440583    5428 patch_prober.go:29] interesting pod/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 29 23:32:59.440824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:59.440639    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t" podUID="db63574f-0719-4902-989b-aac4b2d5c994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 29 23:32:59.579826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:32:59.573315    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:00.037297 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:00.037262    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:00.255192 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:00.255152    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:00.440876 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:00.440797    5428 patch_prober.go:29] interesting pod/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 29 23:33:00.441064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:00.440904    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t" podUID="db63574f-0719-4902-989b-aac4b2d5c994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 29 23:33:00.558486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:00.558456    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:01.402960 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:01.402929    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:01.440624 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:01.440593    5428 patch_prober.go:29] interesting pod/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 29 23:33:01.440833 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:01.440643    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t" podUID="db63574f-0719-4902-989b-aac4b2d5c994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 29 23:33:02.441652 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:02.441613    5428 patch_prober.go:29] interesting pod/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 29 23:33:02.442168 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:02.441698    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t" podUID="db63574f-0719-4902-989b-aac4b2d5c994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 29 23:33:02.513630 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:02.513587    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:03.441584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:03.441541    5428 patch_prober.go:29] interesting pod/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 29 23:33:03.441825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:03.441611    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t" podUID="db63574f-0719-4902-989b-aac4b2d5c994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 29 23:33:04.372980 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:04.372923    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:04.447166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:04.447128    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:33:04.447689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:04.447668    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:33:04.448609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:04.448577    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:33:04.490750 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:04.490696    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:04.673881 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:04.611650    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:06.038648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:06.038599    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:06.169792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:06.169760    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:06.317003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:06.305781    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w"]
Oct 29 23:33:06.397824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:06.397759    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w"]
Oct 29 23:33:06.398254 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:06.398152    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w"
Oct 29 23:33:06.420853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:06.418659    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9965bf-dockercfg-j9llb\""
Oct 29 23:33:06.420853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:06.418783    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9965bf\""
Oct 29 23:33:06.584853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:06.584742    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llnfs\" (UniqueName: \"kubernetes.io/projected/c784cb9f-4716-4092-bf8a-5b7637a23a0e-kube-api-access-llnfs\") pod \"payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w\" (UID: \"c784cb9f-4716-4092-bf8a-5b7637a23a0e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w"
Oct 29 23:33:06.585015 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:06.584885    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c784cb9f-4716-4092-bf8a-5b7637a23a0e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w\" (UID: \"c784cb9f-4716-4092-bf8a-5b7637a23a0e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w"
Oct 29 23:33:06.687044 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:06.687009    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llnfs\" (UniqueName: \"kubernetes.io/projected/c784cb9f-4716-4092-bf8a-5b7637a23a0e-kube-api-access-llnfs\") pod \"payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w\" (UID: \"c784cb9f-4716-4092-bf8a-5b7637a23a0e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w"
Oct 29 23:33:06.687238 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:06.687086    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c784cb9f-4716-4092-bf8a-5b7637a23a0e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w\" (UID: \"c784cb9f-4716-4092-bf8a-5b7637a23a0e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w"
Oct 29 23:33:06.687792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:06.687771    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c784cb9f-4716-4092-bf8a-5b7637a23a0e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w\" (UID: \"c784cb9f-4716-4092-bf8a-5b7637a23a0e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w"
Oct 29 23:33:06.710610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:06.710579    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llnfs\" (UniqueName: \"kubernetes.io/projected/c784cb9f-4716-4092-bf8a-5b7637a23a0e-kube-api-access-llnfs\") pod \"payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w\" (UID: \"c784cb9f-4716-4092-bf8a-5b7637a23a0e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w"
Oct 29 23:33:06.788589 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:06.788558    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w"
Oct 29 23:33:07.098996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:07.098919    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w"]
Oct 29 23:33:07.099416 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:33:07.099235    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc784cb9f_4716_4092_bf8a_5b7637a23a0e.slice/crio-9ffde4de7c8f4d97f1518a11d33e284504b8d3c04587914541fe6be28a814fe6 WatchSource:0}: Error finding container 9ffde4de7c8f4d97f1518a11d33e284504b8d3c04587914541fe6be28a814fe6: Status 404 returned error can't find the container with id 9ffde4de7c8f4d97f1518a11d33e284504b8d3c04587914541fe6be28a814fe6
Oct 29 23:33:07.142886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:07.142856    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w" event={"ID":"c784cb9f-4716-4092-bf8a-5b7637a23a0e","Type":"ContainerStarted","Data":"9ffde4de7c8f4d97f1518a11d33e284504b8d3c04587914541fe6be28a814fe6"}
Oct 29 23:33:07.173208 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:07.173181    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:08.153498 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:08.153466    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w" event={"ID":"c784cb9f-4716-4092-bf8a-5b7637a23a0e","Type":"ContainerStarted","Data":"8d0e67deded15c7a2a2e39d2798bd3afbd6fa023b56fa493f8f21d5e9507cd6b"}
Oct 29 23:33:08.153902 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:08.153555    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w"
Oct 29 23:33:08.197313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:08.197260    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w" podStartSLOduration=2.197246745 podStartE2EDuration="2.197246745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:08.194055449 +0000 UTC m=+2991.537004195" watchObservedRunningTime="2026-10-29 23:33:08.197246745 +0000 UTC m=+2991.540195490"
Oct 29 23:33:08.980071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:08.980021    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:10.740656 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:10.740619    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:12.486256 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:12.486213    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:14.294476 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:14.294446    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:16.088319 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:16.088287    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:17.368810 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:33:17.368762    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:33:17.901001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:17.900956    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:19.165076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:19.165046    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w"
Oct 29 23:33:19.736365 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:19.736332    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:19.855777 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:19.855737    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:19.975622 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:19.975582    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:21.671716 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:21.671679    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:23.656863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:23.656768    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:25.981809 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:25.981760    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:28.244088 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:28.244052    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:28.661217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:28.661172    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:28.780074 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:28.780037    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:31.395416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:31.395382    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:31.671235 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:31.671131    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:32.311191 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:32.311144    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:33.542343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:33.542308    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:35.377139 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:35.377096    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:37.178512 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:37.178474    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:39.003654 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:39.003608    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:40.871662 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:40.871625    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:41.365836 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:41.365780    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:41.486910 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:41.486786    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:41.661220 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:41.661137    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:42.067252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:42.067117    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:42.194442 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:42.194408    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:42.352814 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:42.352711    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:42.505504 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:42.505468    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:42.639254 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:42.639223    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:42.797901 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:42.797870    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:43.407562 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:43.407528    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:43.525117 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:43.525085    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:44.872315 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:44.872279    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:44.994426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:44.994373    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:45.111503 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:45.111471    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:46.685088 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:46.685056    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:47.369305 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:33:47.369259    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:33:48.491334 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:48.491298    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:48.879074 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:48.879043    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:49.787913 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:49.787873    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:50.404213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:50.404181    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:52.367567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:52.367532    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:54.587516 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:54.587483    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:55.139550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:55.139514    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:55.613997 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:55.613970    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:56.181365 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:56.181320    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:56.550404 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:56.550270    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:57.149553 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:57.149524    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:33:59.200789 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:33:59.200759    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:01.123033 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:01.123002    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:02.883489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:02.883452    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:04.954582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:04.954550    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:06.984035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:06.983983    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:09.057325 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:09.057283    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:10.318646 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:10.318611    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:10.435191 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:10.435155    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:11.237792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:11.237757    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:12.189789 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:12.189707    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:12.457169 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:12.457087    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:14.028318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:14.028280    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:16.119565 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:16.119500    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:17.369005 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:34:17.368965    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:34:18.018992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:18.018962    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:19.812701 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:19.812669    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:21.592157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.592126    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"]
Oct 29 23:34:21.597252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.597234    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"
Oct 29 23:34:21.613040 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.613010    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9965bf-serving-cert\""
Oct 29 23:34:21.619842 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.619774    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0886ba54-dbe9-4839-af08-89e8c5b47422-metrics-certs\") pod \"maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296\" (UID: \"0886ba54-dbe9-4839-af08-89e8c5b47422\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"
Oct 29 23:34:21.620203 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.620183    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0886ba54-dbe9-4839-af08-89e8c5b47422-maas-api-tls\") pod \"maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296\" (UID: \"0886ba54-dbe9-4839-af08-89e8c5b47422\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"
Oct 29 23:34:21.620410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.620393    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0886ba54-dbe9-4839-af08-89e8c5b47422-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296\" (UID: \"0886ba54-dbe9-4839-af08-89e8c5b47422\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"
Oct 29 23:34:21.620542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.620528    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz5n6\" (UniqueName: \"kubernetes.io/projected/0886ba54-dbe9-4839-af08-89e8c5b47422-kube-api-access-hz5n6\") pod \"maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296\" (UID: \"0886ba54-dbe9-4839-af08-89e8c5b47422\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"
Oct 29 23:34:21.636651 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.636609    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"]
Oct 29 23:34:21.686474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.686443    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:21.722683 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.722647    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0886ba54-dbe9-4839-af08-89e8c5b47422-metrics-certs\") pod \"maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296\" (UID: \"0886ba54-dbe9-4839-af08-89e8c5b47422\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"
Oct 29 23:34:21.722912 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.722784    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0886ba54-dbe9-4839-af08-89e8c5b47422-maas-api-tls\") pod \"maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296\" (UID: \"0886ba54-dbe9-4839-af08-89e8c5b47422\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"
Oct 29 23:34:21.722912 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.722890    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0886ba54-dbe9-4839-af08-89e8c5b47422-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296\" (UID: \"0886ba54-dbe9-4839-af08-89e8c5b47422\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"
Oct 29 23:34:21.723036 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.722931    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz5n6\" (UniqueName: \"kubernetes.io/projected/0886ba54-dbe9-4839-af08-89e8c5b47422-kube-api-access-hz5n6\") pod \"maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296\" (UID: \"0886ba54-dbe9-4839-af08-89e8c5b47422\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"
Oct 29 23:34:21.723600 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.723569    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0886ba54-dbe9-4839-af08-89e8c5b47422-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296\" (UID: \"0886ba54-dbe9-4839-af08-89e8c5b47422\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"
Oct 29 23:34:21.725775 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.725746    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0886ba54-dbe9-4839-af08-89e8c5b47422-metrics-certs\") pod \"maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296\" (UID: \"0886ba54-dbe9-4839-af08-89e8c5b47422\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"
Oct 29 23:34:21.726017 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.725995    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0886ba54-dbe9-4839-af08-89e8c5b47422-maas-api-tls\") pod \"maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296\" (UID: \"0886ba54-dbe9-4839-af08-89e8c5b47422\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"
Oct 29 23:34:21.746629 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.746604    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz5n6\" (UniqueName: \"kubernetes.io/projected/0886ba54-dbe9-4839-af08-89e8c5b47422-kube-api-access-hz5n6\") pod \"maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296\" (UID: \"0886ba54-dbe9-4839-af08-89e8c5b47422\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"
Oct 29 23:34:21.842442 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.842346    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:21.913825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:21.913780    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"
Oct 29 23:34:22.230155 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:22.230123    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"]
Oct 29 23:34:22.231327 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:34:22.231306    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0886ba54_dbe9_4839_af08_89e8c5b47422.slice/crio-b439772dbd1b1b115e3d7901e16f992688b314acd23cfe1a5bdf5ffb162f3a80 WatchSource:0}: Error finding container b439772dbd1b1b115e3d7901e16f992688b314acd23cfe1a5bdf5ffb162f3a80: Status 404 returned error can't find the container with id b439772dbd1b1b115e3d7901e16f992688b314acd23cfe1a5bdf5ffb162f3a80
Oct 29 23:34:22.917961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:22.917858    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296" event={"ID":"0886ba54-dbe9-4839-af08-89e8c5b47422","Type":"ContainerStarted","Data":"7deb798381d28e9480ae3982c47d369fe542196eaa8cd1bba7ec034b404d9835"}
Oct 29 23:34:22.917961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:22.917915    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296" event={"ID":"0886ba54-dbe9-4839-af08-89e8c5b47422","Type":"ContainerStarted","Data":"b439772dbd1b1b115e3d7901e16f992688b314acd23cfe1a5bdf5ffb162f3a80"}
Oct 29 23:34:22.917961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:22.917942    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"
Oct 29 23:34:22.961688 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:22.961630    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296" podStartSLOduration=1.535003604 podStartE2EDuration="1.961616319s" totalImagesPullingTime="426.612715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:34:22.959565765 +0000 UTC m=+3066.302514511" watchObservedRunningTime="2026-10-29 23:34:22.961616319 +0000 UTC m=+3066.304565063"
Oct 29 23:34:23.855621 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:23.855525    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:25.543877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:25.543838    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:27.494916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:27.494879    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:28.932353 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:28.932323    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"
Oct 29 23:34:29.690516 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:29.690486    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:31.492225 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:31.492188    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:31.620389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:31.620354    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:32.294616 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:32.294579    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:33.381451 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:33.381420    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:35.174705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:35.174676    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:37.085813 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:37.085766    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:37.560422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:37.560389    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"]
Oct 29 23:34:37.560617 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:37.560585    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4" podUID="d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" containerName="istio-proxy" containerID="cri-o://6a1eb9df47be2291edee463dbb75f8b1df9999d4e1c79f0493a4dc4d3c22c7ce" gracePeriod=30
Oct 29 23:34:38.786516 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:38.786478    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:40.831319 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:40.831287    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:42.856875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.856846    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 23:34:42.907519 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.907486    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-podinfo\") pod \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") "
Oct 29 23:34:42.907713 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.907540    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-data\") pod \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") "
Oct 29 23:34:42.907713 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.907571    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istiod-ca-cert\") pod \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") "
Oct 29 23:34:42.907713 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.907601    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-envoy\") pod \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") "
Oct 29 23:34:42.907713 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.907700    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-credential-socket\") pod \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") "
Oct 29 23:34:42.907998 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.907835    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-kube-api-access-mkqbk\" (UniqueName: \"kubernetes.io/projected/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-kube-api-access-mkqbk\") pod \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") "
Oct 29 23:34:42.907998 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.907944    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-workload-certs\") pod \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") "
Oct 29 23:34:42.907998 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.907833    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-data" pod "d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" (UID: "d7ce5e15-2b45-4022-9945-04f4a9d3dd5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:42.908162 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.908052    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-token\" (UniqueName: \"kubernetes.io/projected/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-token\") pod \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") "
Oct 29 23:34:42.908162 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.908052    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istiod-ca-cert" pod "d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" (UID: "d7ce5e15-2b45-4022-9945-04f4a9d3dd5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:34:42.908264 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.908185    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-workload-socket\") pod \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\" (UID: \"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d\") "
Oct 29 23:34:42.908264 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.908235    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-credential-socket" pod "d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" (UID: "d7ce5e15-2b45-4022-9945-04f4a9d3dd5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:42.908396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.908283    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-workload-certs" pod "d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" (UID: "d7ce5e15-2b45-4022-9945-04f4a9d3dd5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:42.908517 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.908494    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-workload-socket" pod "d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" (UID: "d7ce5e15-2b45-4022-9945-04f4a9d3dd5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:42.909037 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.909017    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:34:42.909165 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.909038    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:34:42.909165 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.909053    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:34:42.909165 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.909066    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:34:42.909165 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.909079    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:34:42.910110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.910087    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-podinfo" pod "d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" (UID: "d7ce5e15-2b45-4022-9945-04f4a9d3dd5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:34:42.922895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.922854    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-envoy" pod "d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" (UID: "d7ce5e15-2b45-4022-9945-04f4a9d3dd5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:42.923039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.922917    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-token" pod "d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" (UID: "d7ce5e15-2b45-4022-9945-04f4a9d3dd5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:42.923039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:42.922937    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-kube-api-access-mkqbk" pod "d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" (UID: "d7ce5e15-2b45-4022-9945-04f4a9d3dd5d"). InnerVolumeSpecName "kube-api-access-mkqbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:43.010284 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:43.010255    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:34:43.010284 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:43.010283    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:34:43.010284 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:43.010293    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:34:43.010531 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:43.010303    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkqbk\" (UniqueName: \"kubernetes.io/projected/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d-kube-api-access-mkqbk\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:34:43.108531 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:43.108451    5428 generic.go:356] "Generic (PLEG): container finished" podID="d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" containerID="6a1eb9df47be2291edee463dbb75f8b1df9999d4e1c79f0493a4dc4d3c22c7ce" exitCode=0
Oct 29 23:34:43.108531 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:43.108503    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"
Oct 29 23:34:43.108739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:43.108527    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4" event={"ID":"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d","Type":"ContainerDied","Data":"6a1eb9df47be2291edee463dbb75f8b1df9999d4e1c79f0493a4dc4d3c22c7ce"}
Oct 29 23:34:43.108739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:43.108582    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4" event={"ID":"d7ce5e15-2b45-4022-9945-04f4a9d3dd5d","Type":"ContainerDied","Data":"532059e62c9e83d4a103c94e8927273c0df93654f0873a4105ffa3532b183646"}
Oct 29 23:34:43.108739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:43.108601    5428 scope.go:122] "RemoveContainer" containerID="6a1eb9df47be2291edee463dbb75f8b1df9999d4e1c79f0493a4dc4d3c22c7ce"
Oct 29 23:34:43.121835 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:43.121791    5428 scope.go:122] "RemoveContainer" containerID="6a1eb9df47be2291edee463dbb75f8b1df9999d4e1c79f0493a4dc4d3c22c7ce"
Oct 29 23:34:43.122124 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:34:43.122101    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a1eb9df47be2291edee463dbb75f8b1df9999d4e1c79f0493a4dc4d3c22c7ce\": container with ID starting with 6a1eb9df47be2291edee463dbb75f8b1df9999d4e1c79f0493a4dc4d3c22c7ce not found: ID does not exist" containerID="6a1eb9df47be2291edee463dbb75f8b1df9999d4e1c79f0493a4dc4d3c22c7ce"
Oct 29 23:34:43.122211 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:43.122132    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a1eb9df47be2291edee463dbb75f8b1df9999d4e1c79f0493a4dc4d3c22c7ce"} err="failed to get container status \"6a1eb9df47be2291edee463dbb75f8b1df9999d4e1c79f0493a4dc4d3c22c7ce\": rpc error: code = NotFound desc = could not find container \"6a1eb9df47be2291edee463dbb75f8b1df9999d4e1c79f0493a4dc4d3c22c7ce\": container with ID starting with 6a1eb9df47be2291edee463dbb75f8b1df9999d4e1c79f0493a4dc4d3c22c7ce not found: ID does not exist"
Oct 29 23:34:43.146421 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:43.146401    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:43.165542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:43.165517    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"]
Oct 29 23:34:43.170294 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:43.170272    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-57fm4"]
Oct 29 23:34:44.780226 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:44.780190    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" path="/var/lib/kubelet/pods/d7ce5e15-2b45-4022-9945-04f4a9d3dd5d/volumes"
Oct 29 23:34:45.174224 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:45.174197    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:45.300632 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:45.300599    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:45.657754 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:45.657699    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:46.461816 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:46.461767    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:47.110747 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:47.110715    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:47.370484 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:34:47.370447    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:34:47.456245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:47.456219    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:47.590793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:47.590756    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:47.711002 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:47.710918    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:49.503312 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:49.503275    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:51.511994 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:51.511957    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:53.564646 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:53.564612    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:55.526759 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:55.526724    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:57.656590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:57.656562    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:59.735664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:59.735629    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:34:59.858730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:34:59.858694    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:00.028525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:00.028417    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:01.537193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:01.537152    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:01.690461 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:01.690418    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:01.738103 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:01.738067    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq"]
Oct 29 23:35:01.739138 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:01.739115    5428 cpu_manager.go:414] "containerMap: removing container" podUID="d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" containerName="istio-proxy"
Oct 29 23:35:01.739138 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:01.739137    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" containerName="istio-proxy"
Oct 29 23:35:01.739382 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:01.739367    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7ce5e15-2b45-4022-9945-04f4a9d3dd5d" containerName="istio-proxy"
Oct 29 23:35:01.743875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:01.743847    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq"
Oct 29 23:35:01.764001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:01.763974    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq"]
Oct 29 23:35:01.769293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:01.769271    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b7a567-dockercfg-75cgk\""
Oct 29 23:35:01.789819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:01.789735    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b7a567\""
Oct 29 23:35:01.822232 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:01.822197    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvqgc\" (UniqueName: \"kubernetes.io/projected/3ab912fe-021d-406d-8120-c428d151f1d8-kube-api-access-bvqgc\") pod \"payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq\" (UID: \"3ab912fe-021d-406d-8120-c428d151f1d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq"
Oct 29 23:35:01.822537 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:01.822512    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ab912fe-021d-406d-8120-c428d151f1d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq\" (UID: \"3ab912fe-021d-406d-8120-c428d151f1d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq"
Oct 29 23:35:01.830161 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:01.830132    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:01.926826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:01.925849    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ab912fe-021d-406d-8120-c428d151f1d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq\" (UID: \"3ab912fe-021d-406d-8120-c428d151f1d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq"
Oct 29 23:35:01.926826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:01.926080    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvqgc\" (UniqueName: \"kubernetes.io/projected/3ab912fe-021d-406d-8120-c428d151f1d8-kube-api-access-bvqgc\") pod \"payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq\" (UID: \"3ab912fe-021d-406d-8120-c428d151f1d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq"
Oct 29 23:35:01.928317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:01.927269    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ab912fe-021d-406d-8120-c428d151f1d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq\" (UID: \"3ab912fe-021d-406d-8120-c428d151f1d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq"
Oct 29 23:35:01.962466 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:01.961949    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvqgc\" (UniqueName: \"kubernetes.io/projected/3ab912fe-021d-406d-8120-c428d151f1d8-kube-api-access-bvqgc\") pod \"payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq\" (UID: \"3ab912fe-021d-406d-8120-c428d151f1d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq"
Oct 29 23:35:02.063072 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:02.062978    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq"
Oct 29 23:35:02.433285 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:02.433249    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq"]
Oct 29 23:35:02.439770 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:35:02.439745    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab912fe_021d_406d_8120_c428d151f1d8.slice/crio-70078a140edc491d80fc8e2d97f8cf1449462520ac2cadc99dab778edbc78207 WatchSource:0}: Error finding container 70078a140edc491d80fc8e2d97f8cf1449462520ac2cadc99dab778edbc78207: Status 404 returned error can't find the container with id 70078a140edc491d80fc8e2d97f8cf1449462520ac2cadc99dab778edbc78207
Oct 29 23:35:03.334445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:03.334411    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq" event={"ID":"3ab912fe-021d-406d-8120-c428d151f1d8","Type":"ContainerStarted","Data":"1f0ba8924c9d243ada5da71676ca42cbf0a76ad86bb106db7f70fa5ed00ad37a"}
Oct 29 23:35:03.335036 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:03.334473    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq"
Oct 29 23:35:03.335036 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:03.334493    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq" event={"ID":"3ab912fe-021d-406d-8120-c428d151f1d8","Type":"ContainerStarted","Data":"70078a140edc491d80fc8e2d97f8cf1449462520ac2cadc99dab778edbc78207"}
Oct 29 23:35:03.387011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:03.386943    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq" podStartSLOduration=2.386926818 podStartE2EDuration="2.386926818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:35:03.382313951 +0000 UTC m=+3106.725262710" watchObservedRunningTime="2026-10-29 23:35:03.386926818 +0000 UTC m=+3106.729875562"
Oct 29 23:35:04.095156 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:04.095097    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:05.883006 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:05.882968    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:08.176110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:08.176076    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:09.910315 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:09.910252    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:11.779306 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:11.779278    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:13.581271 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:13.581238    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:14.347222 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:14.347193    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq"
Oct 29 23:35:15.407945 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:15.407914    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:17.249456 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:17.249424    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:17.371764 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:35:17.371728    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:35:19.075909 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:19.075869    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:20.885972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:20.885933    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:22.620097 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:22.620063    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:24.338437 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:24.338361    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"]
Oct 29 23:35:24.345107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:24.345082    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:35:24.364876 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:24.364847    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"]
Oct 29 23:35:24.379845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:24.371123    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6d9e36-serving-cert\""
Oct 29 23:35:24.400864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:24.400793    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjftp\" (UniqueName: \"kubernetes.io/projected/d2b09df9-9a85-4fca-b1db-e17772dd738d-kube-api-access-kjftp\") pod \"maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww\" (UID: \"d2b09df9-9a85-4fca-b1db-e17772dd738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:35:24.401052 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:24.400913    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2b09df9-9a85-4fca-b1db-e17772dd738d-metrics-certs\") pod \"maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww\" (UID: \"d2b09df9-9a85-4fca-b1db-e17772dd738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:35:24.401128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:24.401083    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b09df9-9a85-4fca-b1db-e17772dd738d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww\" (UID: \"d2b09df9-9a85-4fca-b1db-e17772dd738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:35:24.401194 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:24.401176    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b09df9-9a85-4fca-b1db-e17772dd738d-maas-api-tls\") pod \"maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww\" (UID: \"d2b09df9-9a85-4fca-b1db-e17772dd738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:35:24.436337 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:24.436298    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:24.502516 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:24.502487    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2b09df9-9a85-4fca-b1db-e17772dd738d-metrics-certs\") pod \"maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww\" (UID: \"d2b09df9-9a85-4fca-b1db-e17772dd738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:35:24.502673 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:24.502613    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b09df9-9a85-4fca-b1db-e17772dd738d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww\" (UID: \"d2b09df9-9a85-4fca-b1db-e17772dd738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:35:24.502719 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:24.502670    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b09df9-9a85-4fca-b1db-e17772dd738d-maas-api-tls\") pod \"maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww\" (UID: \"d2b09df9-9a85-4fca-b1db-e17772dd738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:35:24.502793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:24.502777    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjftp\" (UniqueName: \"kubernetes.io/projected/d2b09df9-9a85-4fca-b1db-e17772dd738d-kube-api-access-kjftp\") pod \"maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww\" (UID: \"d2b09df9-9a85-4fca-b1db-e17772dd738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:35:24.502934 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:35:24.502781    5428 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-serving-cert: secret "maas-api-e2e-inf-b-6d9e36-serving-cert" not found
Oct 29 23:35:24.503019 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:35:24.503004    5428 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2b09df9-9a85-4fca-b1db-e17772dd738d-maas-api-tls podName:d2b09df9-9a85-4fca-b1db-e17772dd738d nodeName:}" failed. No retries permitted until 2026-10-29 23:35:25.002978868 +0000 UTC m=+3128.345927596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2b09df9-9a85-4fca-b1db-e17772dd738d-maas-api-tls") pod "maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww" (UID: "d2b09df9-9a85-4fca-b1db-e17772dd738d") : secret "maas-api-e2e-inf-b-6d9e36-serving-cert" not found
Oct 29 23:35:24.503318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:24.503301    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b09df9-9a85-4fca-b1db-e17772dd738d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww\" (UID: \"d2b09df9-9a85-4fca-b1db-e17772dd738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:35:24.505131 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:24.505105    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2b09df9-9a85-4fca-b1db-e17772dd738d-metrics-certs\") pod \"maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww\" (UID: \"d2b09df9-9a85-4fca-b1db-e17772dd738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:35:24.531016 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:24.530707    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjftp\" (UniqueName: \"kubernetes.io/projected/d2b09df9-9a85-4fca-b1db-e17772dd738d-kube-api-access-kjftp\") pod \"maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww\" (UID: \"d2b09df9-9a85-4fca-b1db-e17772dd738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:35:24.569740 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:24.569710    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:24.827062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:24.827025    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:25.010703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:25.010666    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b09df9-9a85-4fca-b1db-e17772dd738d-maas-api-tls\") pod \"maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww\" (UID: \"d2b09df9-9a85-4fca-b1db-e17772dd738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:35:25.013249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:25.013223    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b09df9-9a85-4fca-b1db-e17772dd738d-maas-api-tls\") pod \"maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww\" (UID: \"d2b09df9-9a85-4fca-b1db-e17772dd738d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:35:25.262023 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:25.261988    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:35:25.607188 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:25.607157    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"]
Oct 29 23:35:25.608443 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:35:25.608405    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b09df9_9a85_4fca_b1db_e17772dd738d.slice/crio-c24423670cd560c38e2ec441e58079cc1f31d910eb7b0779b07ff27a1bf5dcca WatchSource:0}: Error finding container c24423670cd560c38e2ec441e58079cc1f31d910eb7b0779b07ff27a1bf5dcca: Status 404 returned error can't find the container with id c24423670cd560c38e2ec441e58079cc1f31d910eb7b0779b07ff27a1bf5dcca
Oct 29 23:35:26.294989 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:26.294909    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:26.599621 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:26.599526    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww" event={"ID":"d2b09df9-9a85-4fca-b1db-e17772dd738d","Type":"ContainerStarted","Data":"c458b5f7a42e8bfa9caa38e8fa3b4d2d1750a5565676b04bc3284a4f884836fb"}
Oct 29 23:35:26.599621 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:26.599610    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww" event={"ID":"d2b09df9-9a85-4fca-b1db-e17772dd738d","Type":"ContainerStarted","Data":"c24423670cd560c38e2ec441e58079cc1f31d910eb7b0779b07ff27a1bf5dcca"}
Oct 29 23:35:26.599905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:26.599635    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:35:26.642389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:26.642338    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww" podStartSLOduration=2.272163819 podStartE2EDuration="2.642324508s" totalImagesPullingTime="370.160689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:35:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:35:26.640764332 +0000 UTC m=+3129.983713077" watchObservedRunningTime="2026-10-29 23:35:26.642324508 +0000 UTC m=+3129.985273276"
Oct 29 23:35:28.451588 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:28.451554    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:30.622482 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:30.622444    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:31.622860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:31.622828    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:32.366999 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:32.366966    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:32.476671 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:32.476624    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:32.615223 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:32.615190    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:35:34.194340 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:34.194302    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:35.848470 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:35.848433    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:37.594176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:37.594141    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:39.739098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:39.739071    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:41.765239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:41.765201    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:43.313006 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:43.312951    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:43.567002 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:43.566921    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:45.247373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:45.247337    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:46.268281 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:46.268244    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:47.074006 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:47.073972    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:47.370095 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:35:47.370056    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:35:48.885477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:48.885445    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:50.483959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:50.483929    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:52.167379 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:52.167341    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:53.905726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:53.905692    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:55.730053 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:55.730018    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:57.415722 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:57.415685    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:35:59.110363 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:35:59.110324    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:00.736706 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:00.736666    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:02.745360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:02.745323    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:04.614583 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:04.614548    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:06.484869 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:06.484793    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:08.314381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:08.314342    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:10.019762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:10.019724    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:11.833762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:11.833728    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:11.946759 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:11.946712    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:13.480061 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:13.480025    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:14.016369 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:14.016338    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:15.756081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:15.756052    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:17.370974 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:36:17.370931    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:36:17.563707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:17.563642    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:19.511687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:19.511653    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:21.283032 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:21.282989    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:23.071098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:23.071054    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:24.771452 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:24.771424    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:26.471857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:26.471825    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:28.265291 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:28.265244    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:29.896920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:29.896885    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:31.210431 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:31.210399    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:31.672104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:31.672075    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:31.783785 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:31.783754    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:32.532955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:32.532912    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:33.371665 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:33.371636    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:34.989331 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:34.989294    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:36.617865 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:36.617783    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:38.820772 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:38.820733    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:40.570349 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:40.570311    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:42.465846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:42.465795    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:43.729727 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:43.729689    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:44.808822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:44.806894    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:46.626557 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:46.626515    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:47.371727 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:36:47.371689    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:36:48.346735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:48.346695    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:48.487446 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:48.487405    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:50.478104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:50.478056    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:52.165524 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:52.165488    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:53.884117 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:53.884077    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:55.565450 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:55.565388    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:57.258080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:57.258036    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:36:58.967420 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:36:58.967388    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:00.674219 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:00.674186    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:02.568065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:02.568033    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:04.296245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:04.296207    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:06.159062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:06.159023    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:07.824985 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:07.824949    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:09.594747 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:09.594707    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:09.722687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:09.722565    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:09.838279 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:09.838244    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:10.543768 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:10.543730    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:11.995588 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:11.995552    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:13.799674 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:13.799628    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:15.487519 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:15.487484    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:17.096945 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:17.096903    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:17.237692 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:17.237656    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:17.385463 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:37:17.385387    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:37:18.854232 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:18.854195    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:20.545116 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:20.545059    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:20.682130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:20.682088    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:20.819923 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:20.819839    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:20.947662 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:20.947625    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:22.017593 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.017560    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:22.153355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.153322    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:22.561065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.561031    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"]
Oct 29 23:37:22.566353 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.566330    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"
Oct 29 23:37:22.589644 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.589608    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"]
Oct 29 23:37:22.594103 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.594071    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"agc-cred-40f288853eea\""
Oct 29 23:37:22.594269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.594131    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"payload-processing-external-model-plugins-e2e-extmod-b-206095\""
Oct 29 23:37:22.594269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.594159    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"openshift-service-ca.crt\""
Oct 29 23:37:22.594269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.594163    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"kube-root-ca.crt\""
Oct 29 23:37:22.596318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.596295    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"routing-overlay\""
Oct 29 23:37:22.596468 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.596404    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-206095\"/\"payload-processing-external-model-e2e--dc95ba64-dockercfg-7x46p\""
Oct 29 23:37:22.729949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.729919    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e729ed44-0a10-4647-beb4-0a4c8ca9108b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46\" (UID: \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"
Oct 29 23:37:22.730123 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.730066    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e729ed44-0a10-4647-beb4-0a4c8ca9108b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46\" (UID: \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"
Oct 29 23:37:22.730192 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.730177    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e729ed44-0a10-4647-beb4-0a4c8ca9108b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46\" (UID: \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"
Oct 29 23:37:22.730296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.730279    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4xd8\" (UniqueName: \"kubernetes.io/projected/e729ed44-0a10-4647-beb4-0a4c8ca9108b-kube-api-access-x4xd8\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46\" (UID: \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"
Oct 29 23:37:22.832105 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.832024    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e729ed44-0a10-4647-beb4-0a4c8ca9108b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46\" (UID: \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"
Oct 29 23:37:22.832302 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.832136    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4xd8\" (UniqueName: \"kubernetes.io/projected/e729ed44-0a10-4647-beb4-0a4c8ca9108b-kube-api-access-x4xd8\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46\" (UID: \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"
Oct 29 23:37:22.832302 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.832278    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e729ed44-0a10-4647-beb4-0a4c8ca9108b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46\" (UID: \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"
Oct 29 23:37:22.832437 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.832414    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e729ed44-0a10-4647-beb4-0a4c8ca9108b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46\" (UID: \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"
Oct 29 23:37:22.832948 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.832925    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e729ed44-0a10-4647-beb4-0a4c8ca9108b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46\" (UID: \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"
Oct 29 23:37:22.833069 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.832971    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e729ed44-0a10-4647-beb4-0a4c8ca9108b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46\" (UID: \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"
Oct 29 23:37:22.834975 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.834956    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e729ed44-0a10-4647-beb4-0a4c8ca9108b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46\" (UID: \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"
Oct 29 23:37:22.855056 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.855032    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4xd8\" (UniqueName: \"kubernetes.io/projected/e729ed44-0a10-4647-beb4-0a4c8ca9108b-kube-api-access-x4xd8\") pod \"payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46\" (UID: \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\") " pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"
Oct 29 23:37:22.881444 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:22.881421    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"
Oct 29 23:37:23.062776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:23.062744    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:23.186131 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:23.186097    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:23.205595 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:23.205561    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"]
Oct 29 23:37:23.214061 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:37:23.214028    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode729ed44_0a10_4647_beb4_0a4c8ca9108b.slice/crio-12d1c607217faf49a53f2ef6d67bdedac0a7fa3aaa418411a3a16e78afe86c6b WatchSource:0}: Error finding container 12d1c607217faf49a53f2ef6d67bdedac0a7fa3aaa418411a3a16e78afe86c6b: Status 404 returned error can't find the container with id 12d1c607217faf49a53f2ef6d67bdedac0a7fa3aaa418411a3a16e78afe86c6b
Oct 29 23:37:23.673690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:23.673650    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:23.761139 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:23.761096    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46" event={"ID":"e729ed44-0a10-4647-beb4-0a4c8ca9108b","Type":"ContainerStarted","Data":"f67cc1b69ed77fb0a87db5abad7c3fe2e5d35be407627ea9c5389c4b1b07eddc"}
Oct 29 23:37:23.761735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:23.761169    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"
Oct 29 23:37:23.761735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:23.761187    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46" event={"ID":"e729ed44-0a10-4647-beb4-0a4c8ca9108b","Type":"ContainerStarted","Data":"12d1c607217faf49a53f2ef6d67bdedac0a7fa3aaa418411a3a16e78afe86c6b"}
Oct 29 23:37:23.790035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:23.790003    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:23.803109 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:23.803054    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46" podStartSLOduration=1.8030399959999999 podStartE2EDuration="1.803039996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:23.802297916 +0000 UTC m=+3247.145246662" watchObservedRunningTime="2026-10-29 23:37:23.803039996 +0000 UTC m=+3247.145988740"
Oct 29 23:37:25.372020 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:25.371984    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:25.507025 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:25.506972    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:27.448218 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:27.448185    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:27.569054 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:27.569020    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:27.927364 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:27.927323    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:28.041633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:28.041602    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:28.435380 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:28.435346    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:28.589834 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:28.589271    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:29.919294 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:29.919255    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:31.270205 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:31.270166    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:31.392627 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:31.392595    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:31.516468 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:31.516435    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:31.676466 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:31.676432    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:31.796977 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:31.796944    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:31.923758 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:31.923725    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:32.262398 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:32.261794    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:32.400757 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:32.400720    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:32.634589 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:32.634540    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:33.639156 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:33.639120    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:33.918436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:33.918316    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:34.110481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:34.110445    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:34.185093 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:34.185000    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69"]
Oct 29 23:37:34.190067 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:34.190041    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69"
Oct 29 23:37:34.210573 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:34.210536    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69"]
Oct 29 23:37:34.251934 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:34.249099    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a8ab3c\""
Oct 29 23:37:34.251934 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:34.249261    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a8ab3c-dockercfg-2f9rp\""
Oct 29 23:37:34.374822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:34.355023    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czkxz\" (UniqueName: \"kubernetes.io/projected/69d00263-4608-4899-bc32-f3a966213cb9-kube-api-access-czkxz\") pod \"payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69\" (UID: \"69d00263-4608-4899-bc32-f3a966213cb9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69"
Oct 29 23:37:34.394669 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:34.375709    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d00263-4608-4899-bc32-f3a966213cb9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69\" (UID: \"69d00263-4608-4899-bc32-f3a966213cb9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69"
Oct 29 23:37:34.479309 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:34.479220    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d00263-4608-4899-bc32-f3a966213cb9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69\" (UID: \"69d00263-4608-4899-bc32-f3a966213cb9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69"
Oct 29 23:37:34.479560 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:34.479539    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkxz\" (UniqueName: \"kubernetes.io/projected/69d00263-4608-4899-bc32-f3a966213cb9-kube-api-access-czkxz\") pod \"payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69\" (UID: \"69d00263-4608-4899-bc32-f3a966213cb9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69"
Oct 29 23:37:34.480197 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:34.480174    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d00263-4608-4899-bc32-f3a966213cb9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69\" (UID: \"69d00263-4608-4899-bc32-f3a966213cb9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69"
Oct 29 23:37:34.503091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:34.503065    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czkxz\" (UniqueName: \"kubernetes.io/projected/69d00263-4608-4899-bc32-f3a966213cb9-kube-api-access-czkxz\") pod \"payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69\" (UID: \"69d00263-4608-4899-bc32-f3a966213cb9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69"
Oct 29 23:37:34.528170 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:34.528140    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69"
Oct 29 23:37:34.782656 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:34.782582    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"
Oct 29 23:37:34.879607 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:34.879570    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69"]
Oct 29 23:37:34.918106 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:34.918072    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69" event={"ID":"69d00263-4608-4899-bc32-f3a966213cb9","Type":"ContainerStarted","Data":"8d0603584a7b39866af31cc39e50b06d0653f3d9c6cb25a2654a024ef9653fbe"}
Oct 29 23:37:35.676225 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:35.676186    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:35.928060 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:35.927972    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69" event={"ID":"69d00263-4608-4899-bc32-f3a966213cb9","Type":"ContainerStarted","Data":"ca0bc6d633e463c52516cca75034ed4128b5438bb5208b97075a045984f61d07"}
Oct 29 23:37:35.928425 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:35.928094    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69"
Oct 29 23:37:35.971364 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:35.971320    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69" podStartSLOduration=1.971305868 podStartE2EDuration="1.971305868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:35.968030109 +0000 UTC m=+3259.310978855" watchObservedRunningTime="2026-10-29 23:37:35.971305868 +0000 UTC m=+3259.314254614"
Oct 29 23:37:37.468492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:37.468462    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:37.753296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:37.753193    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:37.871732 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:37.871700    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:38.244792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:38.244752    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:38.846440 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:38.846347    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:39.653311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:39.653282    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:39.814561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:39.814521    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:41.520902 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:41.520870    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:43.512428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:43.512389    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:43.735689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:43.735652    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:45.108313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:45.108280    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:45.358107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:45.357849    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:45.371638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:45.371607    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"]
Oct 29 23:37:45.533146 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:45.533107    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:45.791899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:45.791754    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:46.215418 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:46.215316    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:46.915498 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:46.915464    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:46.942795 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:46.942768    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69"
Oct 29 23:37:47.370101 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:37:47.370068    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:37:47.529445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:47.529411    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:47.683921 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:47.683792    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:49.132213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:49.132183    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:50.373382 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:50.373344    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46" podUID="e729ed44-0a10-4647-beb4-0a4c8ca9108b" containerName="payload-processing" containerID="cri-o://f67cc1b69ed77fb0a87db5abad7c3fe2e5d35be407627ea9c5389c4b1b07eddc" gracePeriod=25
Oct 29 23:37:50.672743 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:50.672717    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"
Oct 29 23:37:50.757203 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:50.757162    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e729ed44-0a10-4647-beb4-0a4c8ca9108b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e729ed44-0a10-4647-beb4-0a4c8ca9108b-provider-credentials\") pod \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\" (UID: \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\") "
Oct 29 23:37:50.757386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:50.757329    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e729ed44-0a10-4647-beb4-0a4c8ca9108b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e729ed44-0a10-4647-beb4-0a4c8ca9108b-routing-overlay-volume\") pod \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\" (UID: \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\") "
Oct 29 23:37:50.757484 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:50.757396    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e729ed44-0a10-4647-beb4-0a4c8ca9108b-kube-api-access-x4xd8\" (UniqueName: \"kubernetes.io/projected/e729ed44-0a10-4647-beb4-0a4c8ca9108b-kube-api-access-x4xd8\") pod \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\" (UID: \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\") "
Oct 29 23:37:50.757550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:50.757516    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e729ed44-0a10-4647-beb4-0a4c8ca9108b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e729ed44-0a10-4647-beb4-0a4c8ca9108b-plugins-config-volume\") pod \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\" (UID: \"e729ed44-0a10-4647-beb4-0a4c8ca9108b\") "
Oct 29 23:37:50.757941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:50.757724    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e729ed44-0a10-4647-beb4-0a4c8ca9108b-routing-overlay-volume" pod "e729ed44-0a10-4647-beb4-0a4c8ca9108b" (UID: "e729ed44-0a10-4647-beb4-0a4c8ca9108b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:37:50.758315 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:50.758281    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e729ed44-0a10-4647-beb4-0a4c8ca9108b-plugins-config-volume" pod "e729ed44-0a10-4647-beb4-0a4c8ca9108b" (UID: "e729ed44-0a10-4647-beb4-0a4c8ca9108b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:37:50.758431 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:50.758358    5428 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e729ed44-0a10-4647-beb4-0a4c8ca9108b-routing-overlay-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:37:50.760057 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:50.760029    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e729ed44-0a10-4647-beb4-0a4c8ca9108b-provider-credentials" pod "e729ed44-0a10-4647-beb4-0a4c8ca9108b" (UID: "e729ed44-0a10-4647-beb4-0a4c8ca9108b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:37:50.760214 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:50.760170    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e729ed44-0a10-4647-beb4-0a4c8ca9108b-kube-api-access-x4xd8" pod "e729ed44-0a10-4647-beb4-0a4c8ca9108b" (UID: "e729ed44-0a10-4647-beb4-0a4c8ca9108b"). InnerVolumeSpecName "kube-api-access-x4xd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:37:50.860200 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:50.860168    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4xd8\" (UniqueName: \"kubernetes.io/projected/e729ed44-0a10-4647-beb4-0a4c8ca9108b-kube-api-access-x4xd8\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:37:50.860200 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:50.860196    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e729ed44-0a10-4647-beb4-0a4c8ca9108b-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:37:50.860200 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:50.860207    5428 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e729ed44-0a10-4647-beb4-0a4c8ca9108b-provider-credentials\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:37:50.877174 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:50.877140    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:51.087133 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:51.087053    5428 generic.go:356] "Generic (PLEG): container finished" podID="e729ed44-0a10-4647-beb4-0a4c8ca9108b" containerID="f67cc1b69ed77fb0a87db5abad7c3fe2e5d35be407627ea9c5389c4b1b07eddc" exitCode=0
Oct 29 23:37:51.087133 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:51.087101    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"
Oct 29 23:37:51.087323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:51.087137    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46" event={"ID":"e729ed44-0a10-4647-beb4-0a4c8ca9108b","Type":"ContainerDied","Data":"f67cc1b69ed77fb0a87db5abad7c3fe2e5d35be407627ea9c5389c4b1b07eddc"}
Oct 29 23:37:51.087323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:51.087190    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46" event={"ID":"e729ed44-0a10-4647-beb4-0a4c8ca9108b","Type":"ContainerDied","Data":"12d1c607217faf49a53f2ef6d67bdedac0a7fa3aaa418411a3a16e78afe86c6b"}
Oct 29 23:37:51.087323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:51.087211    5428 scope.go:122] "RemoveContainer" containerID="f67cc1b69ed77fb0a87db5abad7c3fe2e5d35be407627ea9c5389c4b1b07eddc"
Oct 29 23:37:51.098716 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:51.098683    5428 scope.go:122] "RemoveContainer" containerID="f67cc1b69ed77fb0a87db5abad7c3fe2e5d35be407627ea9c5389c4b1b07eddc"
Oct 29 23:37:51.098997 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:37:51.098978    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67cc1b69ed77fb0a87db5abad7c3fe2e5d35be407627ea9c5389c4b1b07eddc\": container with ID starting with f67cc1b69ed77fb0a87db5abad7c3fe2e5d35be407627ea9c5389c4b1b07eddc not found: ID does not exist" containerID="f67cc1b69ed77fb0a87db5abad7c3fe2e5d35be407627ea9c5389c4b1b07eddc"
Oct 29 23:37:51.099057 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:51.099005    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67cc1b69ed77fb0a87db5abad7c3fe2e5d35be407627ea9c5389c4b1b07eddc"} err="failed to get container status \"f67cc1b69ed77fb0a87db5abad7c3fe2e5d35be407627ea9c5389c4b1b07eddc\": rpc error: code = NotFound desc = could not find container \"f67cc1b69ed77fb0a87db5abad7c3fe2e5d35be407627ea9c5389c4b1b07eddc\": container with ID starting with f67cc1b69ed77fb0a87db5abad7c3fe2e5d35be407627ea9c5389c4b1b07eddc not found: ID does not exist"
Oct 29 23:37:51.141905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:51.141870    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"]
Oct 29 23:37:51.146850 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:51.146821    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-206095/payload-processing-external-model-e2e-extmod-b-206095-54bdgqh46"]
Oct 29 23:37:52.416794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:52.416761    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:52.552546 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:52.552512    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:52.792462 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:52.792381    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e729ed44-0a10-4647-beb4-0a4c8ca9108b" path="/var/lib/kubelet/pods/e729ed44-0a10-4647-beb4-0a4c8ca9108b/volumes"
Oct 29 23:37:52.794205 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:52.794177    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:52.919895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:52.919850    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:53.429637 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:53.429606    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:53.617219 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:53.617181    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:54.720910 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:54.720880    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:55.829121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:55.829089    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:55.945256 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:55.945227    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:56.640053 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:56.640019    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:57.654352 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:57.654322    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:57.808003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:57.807974    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:58.528026 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:58.527993    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:59.299367 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:59.299307    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:59.424530 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:59.424500    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:37:59.677822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:59.670586    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn"]
Oct 29 23:37:59.677822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:59.673673    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e729ed44-0a10-4647-beb4-0a4c8ca9108b" containerName="payload-processing"
Oct 29 23:37:59.677822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:59.673696    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e729ed44-0a10-4647-beb4-0a4c8ca9108b" containerName="payload-processing"
Oct 29 23:37:59.677822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:59.674063    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e729ed44-0a10-4647-beb4-0a4c8ca9108b" containerName="payload-processing"
Oct 29 23:37:59.690821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:59.685252    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn"
Oct 29 23:37:59.694258 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:59.694218    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn"]
Oct 29 23:37:59.706633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:59.706607    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-750972-dockercfg-wqf2k\""
Oct 29 23:37:59.723822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:59.713731    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-750972\""
Oct 29 23:37:59.828449 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:59.828410    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e5475e-1e3a-487e-a50c-97851f4e845d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn\" (UID: \"22e5475e-1e3a-487e-a50c-97851f4e845d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn"
Oct 29 23:37:59.828644 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:59.828568    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p84c\" (UniqueName: \"kubernetes.io/projected/22e5475e-1e3a-487e-a50c-97851f4e845d-kube-api-access-6p84c\") pod \"payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn\" (UID: \"22e5475e-1e3a-487e-a50c-97851f4e845d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn"
Oct 29 23:37:59.929609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:59.929523    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p84c\" (UniqueName: \"kubernetes.io/projected/22e5475e-1e3a-487e-a50c-97851f4e845d-kube-api-access-6p84c\") pod \"payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn\" (UID: \"22e5475e-1e3a-487e-a50c-97851f4e845d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn"
Oct 29 23:37:59.929765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:59.929739    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e5475e-1e3a-487e-a50c-97851f4e845d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn\" (UID: \"22e5475e-1e3a-487e-a50c-97851f4e845d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn"
Oct 29 23:37:59.930441 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:59.930421    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e5475e-1e3a-487e-a50c-97851f4e845d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn\" (UID: \"22e5475e-1e3a-487e-a50c-97851f4e845d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn"
Oct 29 23:37:59.953140 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:37:59.953111    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p84c\" (UniqueName: \"kubernetes.io/projected/22e5475e-1e3a-487e-a50c-97851f4e845d-kube-api-access-6p84c\") pod \"payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn\" (UID: \"22e5475e-1e3a-487e-a50c-97851f4e845d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn"
Oct 29 23:38:00.043633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:00.043602    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn"
Oct 29 23:38:00.360780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:00.360750    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn"]
Oct 29 23:38:00.361416 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:38:00.361387    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e5475e_1e3a_487e_a50c_97851f4e845d.slice/crio-ada2ba827d8ffb759d4b6e693dd2b8b88e1dccdaf5698175247b01784dc70788 WatchSource:0}: Error finding container ada2ba827d8ffb759d4b6e693dd2b8b88e1dccdaf5698175247b01784dc70788: Status 404 returned error can't find the container with id ada2ba827d8ffb759d4b6e693dd2b8b88e1dccdaf5698175247b01784dc70788
Oct 29 23:38:00.499382 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:00.499344    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:01.178887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:01.178853    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn" event={"ID":"22e5475e-1e3a-487e-a50c-97851f4e845d","Type":"ContainerStarted","Data":"e396f1ea5cb5d84a2cf589e780a3a939cec7e7de97341e03df45a12b91c78bc0"}
Oct 29 23:38:01.179112 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:01.178909    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn" event={"ID":"22e5475e-1e3a-487e-a50c-97851f4e845d","Type":"ContainerStarted","Data":"ada2ba827d8ffb759d4b6e693dd2b8b88e1dccdaf5698175247b01784dc70788"}
Oct 29 23:38:01.179112 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:01.179074    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn"
Oct 29 23:38:01.221961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:01.221899    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn" podStartSLOduration=2.221886504 podStartE2EDuration="2.221886504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:38:01.21891757 +0000 UTC m=+3284.561866331" watchObservedRunningTime="2026-10-29 23:38:01.221886504 +0000 UTC m=+3284.564835248"
Oct 29 23:38:01.762166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:01.762104    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:02.517477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:02.517422    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:04.266375 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:04.266340    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:05.977028 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:05.976999    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:07.699144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:07.699106    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:09.303914 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:09.303875    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:09.697290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:09.697257    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"]
Oct 29 23:38:09.697485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:09.697464    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt" podUID="72f44773-1ecf-4102-b29b-0b10e5cebaf1" containerName="istio-proxy" containerID="cri-o://2bfd13ece1de9e04d1dd1153cfe05a1d4d7a80730e2c08946899fab2d8e5355a" gracePeriod=30
Oct 29 23:38:11.070281 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:11.070248    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:12.194288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:12.194258    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn"
Oct 29 23:38:12.845198 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:12.845163    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:13.075318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:13.075286    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:38:13.075542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:13.075347    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt" podUID="72f44773-1ecf-4102-b29b-0b10e5cebaf1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:38:14.481733 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:14.481655    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:14.990409 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:14.990381    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:38:15.167224 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.167192    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-envoy\") pod \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") "
Oct 29 23:38:15.167224 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.167229    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-workload-socket\") pod \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") "
Oct 29 23:38:15.167476 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.167252    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-data\") pod \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") "
Oct 29 23:38:15.167476 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.167280    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f44773-1ecf-4102-b29b-0b10e5cebaf1-kube-api-access-p9427\" (UniqueName: \"kubernetes.io/projected/72f44773-1ecf-4102-b29b-0b10e5cebaf1-kube-api-access-p9427\") pod \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") "
Oct 29 23:38:15.167476 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.167372    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-credential-socket\") pod \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") "
Oct 29 23:38:15.167476 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.167411    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-podinfo\") pod \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") "
Oct 29 23:38:15.167711 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.167500    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istiod-ca-cert\") pod \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") "
Oct 29 23:38:15.167711 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.167517    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-workload-socket" pod "72f44773-1ecf-4102-b29b-0b10e5cebaf1" (UID: "72f44773-1ecf-4102-b29b-0b10e5cebaf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:38:15.167711 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.167604    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-data" pod "72f44773-1ecf-4102-b29b-0b10e5cebaf1" (UID: "72f44773-1ecf-4102-b29b-0b10e5cebaf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:38:15.167931 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.167786    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-token\" (UniqueName: \"kubernetes.io/projected/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-token\") pod \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") "
Oct 29 23:38:15.168305 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.168282    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-workload-certs\") pod \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\" (UID: \"72f44773-1ecf-4102-b29b-0b10e5cebaf1\") "
Oct 29 23:38:15.168634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.168326    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-credential-socket" pod "72f44773-1ecf-4102-b29b-0b10e5cebaf1" (UID: "72f44773-1ecf-4102-b29b-0b10e5cebaf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:38:15.168634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.168584    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-workload-certs" pod "72f44773-1ecf-4102-b29b-0b10e5cebaf1" (UID: "72f44773-1ecf-4102-b29b-0b10e5cebaf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:38:15.168844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.168599    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istiod-ca-cert" pod "72f44773-1ecf-4102-b29b-0b10e5cebaf1" (UID: "72f44773-1ecf-4102-b29b-0b10e5cebaf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:38:15.169466 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.169440    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:38:15.169557 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.169476    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:38:15.169557 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.169492    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:38:15.169557 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.169509    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:38:15.169557 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.169523    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:38:15.169865 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.169834    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f44773-1ecf-4102-b29b-0b10e5cebaf1-kube-api-access-p9427" pod "72f44773-1ecf-4102-b29b-0b10e5cebaf1" (UID: "72f44773-1ecf-4102-b29b-0b10e5cebaf1"). InnerVolumeSpecName "kube-api-access-p9427". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:38:15.170085 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.170061    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-envoy" pod "72f44773-1ecf-4102-b29b-0b10e5cebaf1" (UID: "72f44773-1ecf-4102-b29b-0b10e5cebaf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:38:15.170222 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.170203    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-token" pod "72f44773-1ecf-4102-b29b-0b10e5cebaf1" (UID: "72f44773-1ecf-4102-b29b-0b10e5cebaf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:38:15.172552 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.172525    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-podinfo" pod "72f44773-1ecf-4102-b29b-0b10e5cebaf1" (UID: "72f44773-1ecf-4102-b29b-0b10e5cebaf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:38:15.271213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.271180    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:38:15.271213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.271207    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:38:15.271397 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.271218    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9427\" (UniqueName: \"kubernetes.io/projected/72f44773-1ecf-4102-b29b-0b10e5cebaf1-kube-api-access-p9427\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:38:15.271397 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.271228    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f44773-1ecf-4102-b29b-0b10e5cebaf1-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:38:15.314860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.314830    5428 generic.go:356] "Generic (PLEG): container finished" podID="72f44773-1ecf-4102-b29b-0b10e5cebaf1" containerID="2bfd13ece1de9e04d1dd1153cfe05a1d4d7a80730e2c08946899fab2d8e5355a" exitCode=0
Oct 29 23:38:15.315009 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.314875    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"
Oct 29 23:38:15.315009 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.314914    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt" event={"ID":"72f44773-1ecf-4102-b29b-0b10e5cebaf1","Type":"ContainerDied","Data":"2bfd13ece1de9e04d1dd1153cfe05a1d4d7a80730e2c08946899fab2d8e5355a"}
Oct 29 23:38:15.315009 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.314970    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt" event={"ID":"72f44773-1ecf-4102-b29b-0b10e5cebaf1","Type":"ContainerDied","Data":"6867206360ca438357e368774765d8a6ba313b61dbb74e6c8f79d55db2530266"}
Oct 29 23:38:15.315009 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.314997    5428 scope.go:122] "RemoveContainer" containerID="2bfd13ece1de9e04d1dd1153cfe05a1d4d7a80730e2c08946899fab2d8e5355a"
Oct 29 23:38:15.329374 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.329356    5428 scope.go:122] "RemoveContainer" containerID="2bfd13ece1de9e04d1dd1153cfe05a1d4d7a80730e2c08946899fab2d8e5355a"
Oct 29 23:38:15.329678 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:38:15.329649    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bfd13ece1de9e04d1dd1153cfe05a1d4d7a80730e2c08946899fab2d8e5355a\": container with ID starting with 2bfd13ece1de9e04d1dd1153cfe05a1d4d7a80730e2c08946899fab2d8e5355a not found: ID does not exist" containerID="2bfd13ece1de9e04d1dd1153cfe05a1d4d7a80730e2c08946899fab2d8e5355a"
Oct 29 23:38:15.329739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.329691    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfd13ece1de9e04d1dd1153cfe05a1d4d7a80730e2c08946899fab2d8e5355a"} err="failed to get container status \"2bfd13ece1de9e04d1dd1153cfe05a1d4d7a80730e2c08946899fab2d8e5355a\": rpc error: code = NotFound desc = could not find container \"2bfd13ece1de9e04d1dd1153cfe05a1d4d7a80730e2c08946899fab2d8e5355a\": container with ID starting with 2bfd13ece1de9e04d1dd1153cfe05a1d4d7a80730e2c08946899fab2d8e5355a not found: ID does not exist"
Oct 29 23:38:15.370250 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.370211    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"]
Oct 29 23:38:15.382331 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:15.382303    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2591f6-openshift-default-5f4849bdb9-8bdgt"]
Oct 29 23:38:16.154998 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:16.154964    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:16.780487 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:16.780445    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f44773-1ecf-4102-b29b-0b10e5cebaf1" path="/var/lib/kubelet/pods/72f44773-1ecf-4102-b29b-0b10e5cebaf1/volumes"
Oct 29 23:38:17.368719 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:38:17.368678    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:38:17.950420 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:17.950383    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:19.594211 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:19.594176    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:21.362013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:21.361975    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:22.971361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:22.971327    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:24.597369 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:24.597299    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:27.043914 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:27.043882    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:29.842274 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:29.842228    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:31.602301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:31.602258    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:31.997210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:31.997175    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:32.760075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:32.760044    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:33.382344 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:33.382311    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:34.964342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:34.964301    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:36.675370 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:36.675333    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:38.598076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:38.598035    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:40.309509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:40.309470    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:41.994216 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:41.994165    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:43.773405 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:43.773358    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:45.584194 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:45.584159    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:47.255015 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:47.254975    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:47.370492 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:38:47.370326    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:38:47.380959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:47.380934    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:47.496526 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:47.496497    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:49.165275 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:49.165240    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:50.788013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:50.787976    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:52.466625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:52.466586    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:54.170156 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:54.170026    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:54.477602 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:54.477517    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:54.586821 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:54.586778    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:55.167245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:55.167183    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:55.310271 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:55.310226    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:56.461903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:56.461855    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:58.362550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:58.362466    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:38:59.964731 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:38:59.964701    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:00.094312 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:00.094276    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:00.205216 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:00.205183    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:01.456122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:01.456092    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:01.579886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:01.579828    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:01.944030 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:01.943984    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:02.069892 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:02.069859    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:02.449689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:02.449653    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:03.166376 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:03.166347    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:03.822289 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:03.822255    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:06.172832 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:06.172767    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:07.040936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:07.040871    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:07.159433 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:07.159389    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:07.970603 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:07.970566    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:08.721386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:08.721351    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:08.842592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:08.842559    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:08.988909 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:08.988683    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g"]
Oct 29 23:39:08.990981 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:08.990947    5428 cpu_manager.go:414] "containerMap: removing container" podUID="72f44773-1ecf-4102-b29b-0b10e5cebaf1" containerName="istio-proxy"
Oct 29 23:39:08.990981 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:08.990980    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f44773-1ecf-4102-b29b-0b10e5cebaf1" containerName="istio-proxy"
Oct 29 23:39:08.991266 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:08.991247    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f44773-1ecf-4102-b29b-0b10e5cebaf1" containerName="istio-proxy"
Oct 29 23:39:08.997084 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:08.997054    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g"
Oct 29 23:39:09.019743 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:09.019715    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g"]
Oct 29 23:39:09.029111 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:09.027439    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-37498a-dockercfg-ddngq\""
Oct 29 23:39:09.029111 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:09.027582    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-37498a\""
Oct 29 23:39:09.091673 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:09.091625    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b034cafa-0951-4854-899a-a9f03ebb7748-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g\" (UID: \"b034cafa-0951-4854-899a-a9f03ebb7748\") " pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g"
Oct 29 23:39:09.092100 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:09.092062    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dtb8\" (UniqueName: \"kubernetes.io/projected/b034cafa-0951-4854-899a-a9f03ebb7748-kube-api-access-7dtb8\") pod \"payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g\" (UID: \"b034cafa-0951-4854-899a-a9f03ebb7748\") " pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g"
Oct 29 23:39:09.195023 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:09.194786    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b034cafa-0951-4854-899a-a9f03ebb7748-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g\" (UID: \"b034cafa-0951-4854-899a-a9f03ebb7748\") " pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g"
Oct 29 23:39:09.195023 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:09.194875    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dtb8\" (UniqueName: \"kubernetes.io/projected/b034cafa-0951-4854-899a-a9f03ebb7748-kube-api-access-7dtb8\") pod \"payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g\" (UID: \"b034cafa-0951-4854-899a-a9f03ebb7748\") " pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g"
Oct 29 23:39:09.195843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:09.195791    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b034cafa-0951-4854-899a-a9f03ebb7748-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g\" (UID: \"b034cafa-0951-4854-899a-a9f03ebb7748\") " pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g"
Oct 29 23:39:09.224470 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:09.224433    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dtb8\" (UniqueName: \"kubernetes.io/projected/b034cafa-0951-4854-899a-a9f03ebb7748-kube-api-access-7dtb8\") pod \"payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g\" (UID: \"b034cafa-0951-4854-899a-a9f03ebb7748\") " pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g"
Oct 29 23:39:09.373040 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:09.373005    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g"
Oct 29 23:39:09.687390 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:09.687362    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g"]
Oct 29 23:39:09.689744 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:39:09.689712    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb034cafa_0951_4854_899a_a9f03ebb7748.slice/crio-f25d5f2f3d7af7342092f275bdb49ff1a30d217720802fc06c3c49b63973affc WatchSource:0}: Error finding container f25d5f2f3d7af7342092f275bdb49ff1a30d217720802fc06c3c49b63973affc: Status 404 returned error can't find the container with id f25d5f2f3d7af7342092f275bdb49ff1a30d217720802fc06c3c49b63973affc
Oct 29 23:39:09.862923 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:09.862879    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g" event={"ID":"b034cafa-0951-4854-899a-a9f03ebb7748","Type":"ContainerStarted","Data":"708786b8043de9be7e43619270ed12cc4b3222f35c13c28ac96a0cb40f5456b9"}
Oct 29 23:39:09.863156 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:09.862987    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g" event={"ID":"b034cafa-0951-4854-899a-a9f03ebb7748","Type":"ContainerStarted","Data":"f25d5f2f3d7af7342092f275bdb49ff1a30d217720802fc06c3c49b63973affc"}
Oct 29 23:39:09.863156 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:09.863117    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g"
Oct 29 23:39:09.927819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:09.927750    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g" podStartSLOduration=1.927731572 podStartE2EDuration="1.927731572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:39:09.921561186 +0000 UTC m=+3353.264509942" watchObservedRunningTime="2026-10-29 23:39:09.927731572 +0000 UTC m=+3353.270680318"
Oct 29 23:39:10.258341 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:10.258298    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:12.022242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:12.022206    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:13.652773 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:13.652739    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:13.768293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:13.768240    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:13.984899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:13.984793    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:15.382345 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:15.382315    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:17.068601 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:17.068566    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:17.369700 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:39:17.369666    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:39:18.869448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:18.869417    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:18.978924 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:18.978888    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:20.877792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:20.877758    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g"
Oct 29 23:39:21.058346 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:21.058316    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:22.799937 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:22.799902    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:24.472080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:24.472048    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:26.172248 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:26.172218    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:27.444185 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:27.444147    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:27.566666 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:27.566631    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:27.957575 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:27.957541    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:29.635241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:29.635196    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:29.748150 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:29.748110    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:31.387262 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:31.387225    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:32.243573 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:32.243488    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:32.899909 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:32.899872    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:33.167594 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:33.167509    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:34.851482 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:34.851424    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:36.669217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:36.669180    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:38.348149 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:38.348118    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:39.986263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:39.986222    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:40.603269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:40.603222    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:41.964377 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:41.964328    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:43.691617 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:43.691583    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:45.291104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:45.291073    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:47.210599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:47.210553    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:47.369764 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:39:47.369723    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:39:50.322786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:50.322756    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:51.307709 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:51.307676    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:51.428867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:51.428837    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:52.399830 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:52.399787    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:54.184732 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:54.184702    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:56.036637 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:56.036603    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:57.770460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:57.770431    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:39:59.574245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:39:59.574149    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:01.372620 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:01.372578    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:03.195070 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:03.195036    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:03.344178 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:03.344144    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:04.845236 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:04.845198    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:06.617928 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:06.617895    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:08.894054 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:08.894020    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:10.684791 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:10.684753    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:11.195104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:11.195077    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:11.706986 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:11.706928    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:12.190042 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:12.190006    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:12.540880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:12.540736    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:13.075402 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:13.075364    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:14.876068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:14.876023    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:16.002471 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:16.002435    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:16.117626 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:16.117593    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:16.867650 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:16.867614    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:17.372180 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:40:17.372143    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:40:17.721854 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:17.721754    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:17.831210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:17.831178    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:17.982056 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:17.981965    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9"]
Oct 29 23:40:17.993476 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:17.993443    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9"
Oct 29 23:40:18.030471 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:18.030440    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-55c3fc-dockercfg-jq5bv\""
Oct 29 23:40:18.032062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:18.032039    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-55c3fc\""
Oct 29 23:40:18.043335 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:18.043075    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chckp\" (UniqueName: \"kubernetes.io/projected/df3df4b7-f563-47b9-8be3-62f09de6d5fb-kube-api-access-chckp\") pod \"payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9\" (UID: \"df3df4b7-f563-47b9-8be3-62f09de6d5fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9"
Oct 29 23:40:18.043335 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:18.043142    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3df4b7-f563-47b9-8be3-62f09de6d5fb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9\" (UID: \"df3df4b7-f563-47b9-8be3-62f09de6d5fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9"
Oct 29 23:40:18.063903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:18.063870    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9"]
Oct 29 23:40:18.147180 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:18.146510    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chckp\" (UniqueName: \"kubernetes.io/projected/df3df4b7-f563-47b9-8be3-62f09de6d5fb-kube-api-access-chckp\") pod \"payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9\" (UID: \"df3df4b7-f563-47b9-8be3-62f09de6d5fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9"
Oct 29 23:40:18.147180 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:18.146563    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3df4b7-f563-47b9-8be3-62f09de6d5fb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9\" (UID: \"df3df4b7-f563-47b9-8be3-62f09de6d5fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9"
Oct 29 23:40:18.148036 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:18.147990    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3df4b7-f563-47b9-8be3-62f09de6d5fb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9\" (UID: \"df3df4b7-f563-47b9-8be3-62f09de6d5fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9"
Oct 29 23:40:18.172597 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:18.172567    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chckp\" (UniqueName: \"kubernetes.io/projected/df3df4b7-f563-47b9-8be3-62f09de6d5fb-kube-api-access-chckp\") pod \"payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9\" (UID: \"df3df4b7-f563-47b9-8be3-62f09de6d5fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9"
Oct 29 23:40:18.355216 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:18.355126    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9"
Oct 29 23:40:18.678866 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:18.678835    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9"]
Oct 29 23:40:18.679387 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:40:18.679364    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3df4b7_f563_47b9_8be3_62f09de6d5fb.slice/crio-35e2e6bb605a91778e3b5358d343442059031d5f50d4cc39015da9edef1ffcac WatchSource:0}: Error finding container 35e2e6bb605a91778e3b5358d343442059031d5f50d4cc39015da9edef1ffcac: Status 404 returned error can't find the container with id 35e2e6bb605a91778e3b5358d343442059031d5f50d4cc39015da9edef1ffcac
Oct 29 23:40:19.040410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:19.040327    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:19.641597 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:19.641559    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9" event={"ID":"df3df4b7-f563-47b9-8be3-62f09de6d5fb","Type":"ContainerStarted","Data":"73fb9dd75b3db4a49e404688c01cca8f4b3dc20f427f107dd8c07c2b674c6806"}
Oct 29 23:40:19.641776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:19.641626    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9" event={"ID":"df3df4b7-f563-47b9-8be3-62f09de6d5fb","Type":"ContainerStarted","Data":"35e2e6bb605a91778e3b5358d343442059031d5f50d4cc39015da9edef1ffcac"}
Oct 29 23:40:19.641776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:19.641662    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9"
Oct 29 23:40:19.684189 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:19.684139    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9" podStartSLOduration=2.6841252620000002 podStartE2EDuration="2.684125262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:40:19.681232547 +0000 UTC m=+3423.024181292" watchObservedRunningTime="2026-10-29 23:40:19.684125262 +0000 UTC m=+3423.027074005"
Oct 29 23:40:20.762082 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:20.762050    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:22.621620 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:22.621585    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:24.494358 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:24.494327    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:26.096071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:26.096038    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:26.229016 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:26.228966    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:26.358514 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:26.358424    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:27.031004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:27.030953    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:27.495005 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:27.494976    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:27.960825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:27.960776    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:28.282900 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:28.282797    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:28.702180 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:28.702151    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:30.057925 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:30.057882    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:30.655561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:30.655534    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9"
Oct 29 23:40:31.693508 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:31.693469    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:32.257178 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:32.257140    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:33.209815 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:33.209773    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:33.345992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:33.345960    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:35.128432 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:35.128385    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:36.961480 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:36.961448    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:38.601084 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:38.601030    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"]
Oct 29 23:40:38.601581 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:38.601324    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq" podUID="f6420f5b-f272-44f6-86c9-f0f87cafe7ff" containerName="maas-api" containerID="cri-o://a633be99f4b470aa875e562bb4aff8cd5d71ff167154d3222d29df83b906dd2b" gracePeriod=30
Oct 29 23:40:38.669243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:38.669211    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:38.860471 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:38.860367    5428 generic.go:356] "Generic (PLEG): container finished" podID="f6420f5b-f272-44f6-86c9-f0f87cafe7ff" containerID="a633be99f4b470aa875e562bb4aff8cd5d71ff167154d3222d29df83b906dd2b" exitCode=0
Oct 29 23:40:38.860688 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:38.860663    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq" event={"ID":"f6420f5b-f272-44f6-86c9-f0f87cafe7ff","Type":"ContainerDied","Data":"a633be99f4b470aa875e562bb4aff8cd5d71ff167154d3222d29df83b906dd2b"}
Oct 29 23:40:38.963950 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:38.963914    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"
Oct 29 23:40:39.031038 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.030996    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"]
Oct 29 23:40:39.032682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.032648    5428 cpu_manager.go:414] "containerMap: removing container" podUID="f6420f5b-f272-44f6-86c9-f0f87cafe7ff" containerName="maas-api"
Oct 29 23:40:39.032682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.032677    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6420f5b-f272-44f6-86c9-f0f87cafe7ff" containerName="maas-api"
Oct 29 23:40:39.032953 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.032935    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6420f5b-f272-44f6-86c9-f0f87cafe7ff" containerName="maas-api"
Oct 29 23:40:39.038056 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.038034    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.063283 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.063250    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"]
Oct 29 23:40:39.063681 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.063657    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5cd0f2-openshift-default-dockercfg-q2mg6\""
Oct 29 23:40:39.106788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.106753    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-maas-api-tls\") pod \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\" (UID: \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\") "
Oct 29 23:40:39.107021 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.107000    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-metrics-certs\") pod \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\" (UID: \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\") "
Oct 29 23:40:39.107173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.107154    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-openshift-service-ca\") pod \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\" (UID: \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\") "
Oct 29 23:40:39.107279 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.107262    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-kube-api-access-nhclz\" (UniqueName: \"kubernetes.io/projected/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-kube-api-access-nhclz\") pod \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\" (UID: \"f6420f5b-f272-44f6-86c9-f0f87cafe7ff\") "
Oct 29 23:40:39.107557 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.107532    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-openshift-service-ca" pod "f6420f5b-f272-44f6-86c9-f0f87cafe7ff" (UID: "f6420f5b-f272-44f6-86c9-f0f87cafe7ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:40:39.107718 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.107692    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-workload-socket\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.107871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.107852    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-workload-certs\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.107948 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.107911    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd54fa0-a337-47a7-b507-7076286d9963-istio-podinfo\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.108007 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.107995    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd54fa0-a337-47a7-b507-7076286d9963-istio-token\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.108064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.108014    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-istio-data\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.108064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.108043    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-istio-envoy\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.108180 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.108070    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd54fa0-a337-47a7-b507-7076286d9963-istiod-ca-cert\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.108180 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.108133    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pkzc\" (UniqueName: \"kubernetes.io/projected/abd54fa0-a337-47a7-b507-7076286d9963-kube-api-access-2pkzc\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.108292 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.108214    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-credential-socket\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.108292 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.108281    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:40:39.125407 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.116286    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-kube-api-access-nhclz" pod "f6420f5b-f272-44f6-86c9-f0f87cafe7ff" (UID: "f6420f5b-f272-44f6-86c9-f0f87cafe7ff"). InnerVolumeSpecName "kube-api-access-nhclz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:40:39.125407 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.118420    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-maas-api-tls" pod "f6420f5b-f272-44f6-86c9-f0f87cafe7ff" (UID: "f6420f5b-f272-44f6-86c9-f0f87cafe7ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:40:39.125651 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.125403    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-metrics-certs" pod "f6420f5b-f272-44f6-86c9-f0f87cafe7ff" (UID: "f6420f5b-f272-44f6-86c9-f0f87cafe7ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:40:39.209568 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.209526    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-workload-socket\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.209776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.209646    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-workload-certs\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.209776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.209693    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd54fa0-a337-47a7-b507-7076286d9963-istio-podinfo\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.209776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.209750    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd54fa0-a337-47a7-b507-7076286d9963-istio-token\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.209776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.209771    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-istio-data\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.210066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.209837    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-istio-envoy\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.210066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.209880    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd54fa0-a337-47a7-b507-7076286d9963-istiod-ca-cert\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.210066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.209998    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pkzc\" (UniqueName: \"kubernetes.io/projected/abd54fa0-a337-47a7-b507-7076286d9963-kube-api-access-2pkzc\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.210225 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.210114    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-credential-socket\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.210225 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.210134    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-workload-socket\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.210772 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.210691    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-credential-socket\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.210772 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.210731    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:40:39.211457 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.211410    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd54fa0-a337-47a7-b507-7076286d9963-istiod-ca-cert\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.211652 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.211632    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-workload-certs\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.211892 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.211832    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhclz\" (UniqueName: \"kubernetes.io/projected/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-kube-api-access-nhclz\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:40:39.211977 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.211890    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-istio-data\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.211977 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.211909    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6420f5b-f272-44f6-86c9-f0f87cafe7ff-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:40:39.213081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.213056    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-istio-envoy\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.213378 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.213356    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd54fa0-a337-47a7-b507-7076286d9963-istio-podinfo\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.233553 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.233522    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pkzc\" (UniqueName: \"kubernetes.io/projected/abd54fa0-a337-47a7-b507-7076286d9963-kube-api-access-2pkzc\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.234164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.234139    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd54fa0-a337-47a7-b507-7076286d9963-istio-token\") pod \"e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") " pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.372117 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.372082    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:39.643757 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.643676    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:39.709726 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:40:39.709696    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd54fa0_a337_47a7_b507_7076286d9963.slice/crio-be50d49fdbb496d4e0a82736fb56f2a6f84336877501740fcaa710ef0070e8d3 WatchSource:0}: Error finding container be50d49fdbb496d4e0a82736fb56f2a6f84336877501740fcaa710ef0070e8d3: Status 404 returned error can't find the container with id be50d49fdbb496d4e0a82736fb56f2a6f84336877501740fcaa710ef0070e8d3
Oct 29 23:40:39.709903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.709883    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"]
Oct 29 23:40:39.713051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.713019    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:40:39.713137 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.713081    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:40:39.713137 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.713109    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:40:39.764652 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.764626    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:39.873195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.873165    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:39.873394 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.873360    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf" event={"ID":"abd54fa0-a337-47a7-b507-7076286d9963","Type":"ContainerStarted","Data":"6e032b1b732ec57224d54ee1a3f883cc45cc77f9e8d8332f27124a4c198f792e"}
Oct 29 23:40:39.873811 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.873768    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf" event={"ID":"abd54fa0-a337-47a7-b507-7076286d9963","Type":"ContainerStarted","Data":"be50d49fdbb496d4e0a82736fb56f2a6f84336877501740fcaa710ef0070e8d3"}
Oct 29 23:40:39.876121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.876090    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq" event={"ID":"f6420f5b-f272-44f6-86c9-f0f87cafe7ff","Type":"ContainerDied","Data":"d2c2e70d31be5fe16f3a8b9cf80a70a36f6a7b85237aca0d6e59c9211edbab21"}
Oct 29 23:40:39.876366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.876164    5428 scope.go:122] "RemoveContainer" containerID="a633be99f4b470aa875e562bb4aff8cd5d71ff167154d3222d29df83b906dd2b"
Oct 29 23:40:39.877510 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.877044    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"
Oct 29 23:40:39.940883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.940820    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf" podStartSLOduration=21.940788856 podStartE2EDuration="21.940788856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:40:39.927182054 +0000 UTC m=+3443.270130816" watchObservedRunningTime="2026-10-29 23:40:39.940788856 +0000 UTC m=+3443.283737600"
Oct 29 23:40:39.970764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.970731    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"]
Oct 29 23:40:39.975545 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:39.975521    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-90f683-54f8b865cd-fvddq"]
Oct 29 23:40:40.295701 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:40.295629    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:40.372905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:40.372868    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:40.375237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:40.375208    5428 patch_prober.go:29] interesting pod/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 29 23:40:40.375387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:40.375266    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf" podUID="abd54fa0-a337-47a7-b507-7076286d9963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 29 23:40:40.782524 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:40.782485    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6420f5b-f272-44f6-86c9-f0f87cafe7ff" path="/var/lib/kubelet/pods/f6420f5b-f272-44f6-86c9-f0f87cafe7ff/volumes"
Oct 29 23:40:41.379852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:41.378061    5428 patch_prober.go:29] interesting pod/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 29 23:40:41.379852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:41.378125    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf" podUID="abd54fa0-a337-47a7-b507-7076286d9963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 29 23:40:41.653672 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:41.653537    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:41.789791 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:41.789760    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:42.142411 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:42.142374    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:42.382825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:42.382556    5428 patch_prober.go:29] interesting pod/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 29 23:40:42.382825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:42.382626    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf" podUID="abd54fa0-a337-47a7-b507-7076286d9963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 29 23:40:42.684869 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:42.679237    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:43.342842 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:43.342791    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:43.382838 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:43.380179    5428 patch_prober.go:29] interesting pod/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 29 23:40:43.382838 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:43.380259    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf" podUID="abd54fa0-a337-47a7-b507-7076286d9963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 29 23:40:44.379826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:44.378892    5428 patch_prober.go:29] interesting pod/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 29 23:40:44.379826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:44.378963    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf" podUID="abd54fa0-a337-47a7-b507-7076286d9963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 29 23:40:44.737004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:44.736912    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:45.382837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:45.373104    5428 patch_prober.go:29] interesting pod/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 29 23:40:45.382837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:45.373165    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf" podUID="abd54fa0-a337-47a7-b507-7076286d9963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 29 23:40:45.437865 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:45.437826    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:46.376378 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:46.376344    5428 patch_prober.go:29] interesting pod/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 29 23:40:46.376690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:46.376660    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf" podUID="abd54fa0-a337-47a7-b507-7076286d9963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 29 23:40:46.762287 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:46.762196    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:46.900881 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:46.900665    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:47.109941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:47.109280    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"]
Oct 29 23:40:47.109941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:47.109462    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t" podUID="db63574f-0719-4902-989b-aac4b2d5c994" containerName="istio-proxy" containerID="cri-o://fbbb2f6ec9d920466cdd77330c38f3b41a044a2f56712d838538cb014aabaa77" gracePeriod=30
Oct 29 23:40:47.137580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:47.137526    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:47.382074 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:40:47.382035    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:40:47.382294 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:47.382094    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:47.382773 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:47.382700    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:47.383484 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:47.383467    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:40:48.408515 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:48.408476    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:49.450519 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:49.450485    5428 patch_prober.go:29] interesting pod/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:40:49.450923 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:49.450547    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t" podUID="db63574f-0719-4902-989b-aac4b2d5c994" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:40:50.356153 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:50.356105    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:52.042300 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.042254    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:52.160539 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.160505    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:52.403291 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.403267    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:40:52.500754 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.500721    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-credential-socket\") pod \"db63574f-0719-4902-989b-aac4b2d5c994\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") "
Oct 29 23:40:52.501072 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.501040    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db63574f-0719-4902-989b-aac4b2d5c994-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db63574f-0719-4902-989b-aac4b2d5c994-istio-podinfo\") pod \"db63574f-0719-4902-989b-aac4b2d5c994\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") "
Oct 29 23:40:52.501198 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.501129    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-workload-certs\") pod \"db63574f-0719-4902-989b-aac4b2d5c994\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") "
Oct 29 23:40:52.501265 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.501214    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db63574f-0719-4902-989b-aac4b2d5c994-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db63574f-0719-4902-989b-aac4b2d5c994-istiod-ca-cert\") pod \"db63574f-0719-4902-989b-aac4b2d5c994\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") "
Oct 29 23:40:52.501327 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.501043    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-credential-socket" pod "db63574f-0719-4902-989b-aac4b2d5c994" (UID: "db63574f-0719-4902-989b-aac4b2d5c994"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:40:52.501375 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.501332    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-istio-envoy\") pod \"db63574f-0719-4902-989b-aac4b2d5c994\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") "
Oct 29 23:40:52.501375 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.501361    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-istio-data\") pod \"db63574f-0719-4902-989b-aac4b2d5c994\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") "
Oct 29 23:40:52.501472 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.501409    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db63574f-0719-4902-989b-aac4b2d5c994-istio-token\" (UniqueName: \"kubernetes.io/projected/db63574f-0719-4902-989b-aac4b2d5c994-istio-token\") pod \"db63574f-0719-4902-989b-aac4b2d5c994\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") "
Oct 29 23:40:52.501472 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.501457    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-workload-socket\") pod \"db63574f-0719-4902-989b-aac4b2d5c994\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") "
Oct 29 23:40:52.501601 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.501572    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-workload-certs" pod "db63574f-0719-4902-989b-aac4b2d5c994" (UID: "db63574f-0719-4902-989b-aac4b2d5c994"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:40:52.501650 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.501581    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db63574f-0719-4902-989b-aac4b2d5c994-kube-api-access-7hr42\" (UniqueName: \"kubernetes.io/projected/db63574f-0719-4902-989b-aac4b2d5c994-kube-api-access-7hr42\") pod \"db63574f-0719-4902-989b-aac4b2d5c994\" (UID: \"db63574f-0719-4902-989b-aac4b2d5c994\") "
Oct 29 23:40:52.501752 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.501729    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-istio-data" pod "db63574f-0719-4902-989b-aac4b2d5c994" (UID: "db63574f-0719-4902-989b-aac4b2d5c994"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:40:52.501973 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.501950    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db63574f-0719-4902-989b-aac4b2d5c994-istiod-ca-cert" pod "db63574f-0719-4902-989b-aac4b2d5c994" (UID: "db63574f-0719-4902-989b-aac4b2d5c994"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:40:52.502469 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.502447    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:40:52.502550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.502477    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:40:52.502550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.502491    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:40:52.502550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.502504    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db63574f-0719-4902-989b-aac4b2d5c994-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:40:52.502831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.502779    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-workload-socket" pod "db63574f-0719-4902-989b-aac4b2d5c994" (UID: "db63574f-0719-4902-989b-aac4b2d5c994"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:40:52.507945 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.506679    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"]
Oct 29 23:40:52.510883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.508445    5428 cpu_manager.go:414] "containerMap: removing container" podUID="db63574f-0719-4902-989b-aac4b2d5c994" containerName="istio-proxy"
Oct 29 23:40:52.510883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.508472    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db63574f-0719-4902-989b-aac4b2d5c994" containerName="istio-proxy"
Oct 29 23:40:52.510883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.508708    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="db63574f-0719-4902-989b-aac4b2d5c994" containerName="istio-proxy"
Oct 29 23:40:52.514453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.514135    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.520340 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.520316    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"]
Oct 29 23:40:52.522708 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.522531    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db63574f-0719-4902-989b-aac4b2d5c994-istio-podinfo" pod "db63574f-0719-4902-989b-aac4b2d5c994" (UID: "db63574f-0719-4902-989b-aac4b2d5c994"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:40:52.522708 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.522595    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db63574f-0719-4902-989b-aac4b2d5c994-kube-api-access-7hr42" pod "db63574f-0719-4902-989b-aac4b2d5c994" (UID: "db63574f-0719-4902-989b-aac4b2d5c994"). InnerVolumeSpecName "kube-api-access-7hr42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:40:52.522708 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.522681    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db63574f-0719-4902-989b-aac4b2d5c994-istio-token" pod "db63574f-0719-4902-989b-aac4b2d5c994" (UID: "db63574f-0719-4902-989b-aac4b2d5c994"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:40:52.524221 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.524006    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-istio-envoy" pod "db63574f-0719-4902-989b-aac4b2d5c994" (UID: "db63574f-0719-4902-989b-aac4b2d5c994"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:40:52.530522 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.530498    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d20345-openshift-default-dockercfg-cg8t6\""
Oct 29 23:40:52.603528 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.603500    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5hjh\" (UniqueName: \"kubernetes.io/projected/97ec11c4-83fa-4ae5-94fa-35a14eed785f-kube-api-access-h5hjh\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.603667 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.603549    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-data\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.603667 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.603570    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-podinfo\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.603749 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.603711    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-credential-socket\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.603840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.603823    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istiod-ca-cert\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.603895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.603880    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-workload-certs\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.603952 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.603935    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-token\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.604051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.604038    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-workload-socket\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.604110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.604096    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-envoy\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.604361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.604347    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db63574f-0719-4902-989b-aac4b2d5c994-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:40:52.604400 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.604369    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:40:52.604400 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.604384    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db63574f-0719-4902-989b-aac4b2d5c994-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:40:52.604400 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.604396    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db63574f-0719-4902-989b-aac4b2d5c994-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:40:52.604516 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.604411    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hr42\" (UniqueName: \"kubernetes.io/projected/db63574f-0719-4902-989b-aac4b2d5c994-kube-api-access-7hr42\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:40:52.705623 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.705576    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-workload-certs\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.705861 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.705663    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-token\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.705861 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.705735    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-workload-socket\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.705861 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.705836    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-envoy\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.706045 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.706027    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5hjh\" (UniqueName: \"kubernetes.io/projected/97ec11c4-83fa-4ae5-94fa-35a14eed785f-kube-api-access-h5hjh\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.706092 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.706080    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-data\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.706135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.706083    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-workload-certs\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.706135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.706113    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-workload-socket\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.706248 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.706127    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-podinfo\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.706248 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.706196    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-credential-socket\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.706367 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.706273    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istiod-ca-cert\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.706428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.706336    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-data\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.706536 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.706509    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-credential-socket\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.706974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.706954    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istiod-ca-cert\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.708098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.708074    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-envoy\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.708427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.708408    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-podinfo\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.728983 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.728954    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-token\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.729087 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.729034    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5hjh\" (UniqueName: \"kubernetes.io/projected/97ec11c4-83fa-4ae5-94fa-35a14eed785f-kube-api-access-h5hjh\") pod \"e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") " pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:52.865624 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:52.865586    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:53.031262 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:53.031170    5428 generic.go:356] "Generic (PLEG): container finished" podID="db63574f-0719-4902-989b-aac4b2d5c994" containerID="fbbb2f6ec9d920466cdd77330c38f3b41a044a2f56712d838538cb014aabaa77" exitCode=0
Oct 29 23:40:53.031262 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:53.031221    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"
Oct 29 23:40:53.031262 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:53.031253    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t" event={"ID":"db63574f-0719-4902-989b-aac4b2d5c994","Type":"ContainerDied","Data":"fbbb2f6ec9d920466cdd77330c38f3b41a044a2f56712d838538cb014aabaa77"}
Oct 29 23:40:53.031543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:53.031305    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t" event={"ID":"db63574f-0719-4902-989b-aac4b2d5c994","Type":"ContainerDied","Data":"a45f6aaf21e168db759728aec70429d1f0ed1dda54ca2f2ac2ccd5fea79ef624"}
Oct 29 23:40:53.031543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:53.031329    5428 scope.go:122] "RemoveContainer" containerID="fbbb2f6ec9d920466cdd77330c38f3b41a044a2f56712d838538cb014aabaa77"
Oct 29 23:40:53.044296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:53.044269    5428 scope.go:122] "RemoveContainer" containerID="fbbb2f6ec9d920466cdd77330c38f3b41a044a2f56712d838538cb014aabaa77"
Oct 29 23:40:53.044656 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:40:53.044598    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbbb2f6ec9d920466cdd77330c38f3b41a044a2f56712d838538cb014aabaa77\": container with ID starting with fbbb2f6ec9d920466cdd77330c38f3b41a044a2f56712d838538cb014aabaa77 not found: ID does not exist" containerID="fbbb2f6ec9d920466cdd77330c38f3b41a044a2f56712d838538cb014aabaa77"
Oct 29 23:40:53.044656 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:53.044635    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbbb2f6ec9d920466cdd77330c38f3b41a044a2f56712d838538cb014aabaa77"} err="failed to get container status \"fbbb2f6ec9d920466cdd77330c38f3b41a044a2f56712d838538cb014aabaa77\": rpc error: code = NotFound desc = could not find container \"fbbb2f6ec9d920466cdd77330c38f3b41a044a2f56712d838538cb014aabaa77\": container with ID starting with fbbb2f6ec9d920466cdd77330c38f3b41a044a2f56712d838538cb014aabaa77 not found: ID does not exist"
Oct 29 23:40:53.092257 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:53.092070    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"]
Oct 29 23:40:53.097164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:53.097138    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-90f683-openshift-default-859fd4cb87-tl55t"]
Oct 29 23:40:53.198267 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:40:53.198240    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ec11c4_83fa_4ae5_94fa_35a14eed785f.slice/crio-49fc0c3eff47294242e421c2497eb4e4cd185ce51bb7a87cefc94c073b9ae214 WatchSource:0}: Error finding container 49fc0c3eff47294242e421c2497eb4e4cd185ce51bb7a87cefc94c073b9ae214: Status 404 returned error can't find the container with id 49fc0c3eff47294242e421c2497eb4e4cd185ce51bb7a87cefc94c073b9ae214
Oct 29 23:40:53.198822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:53.198782    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"]
Oct 29 23:40:53.202145 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:53.202109    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:40:53.202251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:53.202192    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:40:53.202251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:53.202242    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:40:53.822460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:53.821629    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:54.041466 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:54.041421    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr" event={"ID":"97ec11c4-83fa-4ae5-94fa-35a14eed785f","Type":"ContainerStarted","Data":"59022cec8fa426e35fe84a9f22ea937326357091f5dcdde4b46ad01a08bc558b"}
Oct 29 23:40:54.041674 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:54.041487    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr" event={"ID":"97ec11c4-83fa-4ae5-94fa-35a14eed785f","Type":"ContainerStarted","Data":"49fc0c3eff47294242e421c2497eb4e4cd185ce51bb7a87cefc94c073b9ae214"}
Oct 29 23:40:54.089577 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:54.089459    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr" podStartSLOduration=12.089445564 podStartE2EDuration="12.089445564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:40:54.087330937 +0000 UTC m=+3457.430279683" watchObservedRunningTime="2026-10-29 23:40:54.089445564 +0000 UTC m=+3457.432394308"
Oct 29 23:40:54.779527 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:54.779487    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db63574f-0719-4902-989b-aac4b2d5c994" path="/var/lib/kubelet/pods/db63574f-0719-4902-989b-aac4b2d5c994/volumes"
Oct 29 23:40:54.866301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:54.866267    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:54.873578 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:54.873554    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:55.052250 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:55.052159    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:55.053245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:55.053223    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:40:55.543582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:55.543547    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:57.291916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:57.291885    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:58.409528 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:58.409491    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w"]
Oct 29 23:40:58.553589 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:58.553556    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:58.672912 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:58.672791    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:40:59.011556 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:40:59.011473    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:01.251745 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:01.251702    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:03.410732 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:03.410682    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w" podUID="c784cb9f-4716-4092-bf8a-5b7637a23a0e" containerName="payload-processing" containerID="cri-o://8d0e67deded15c7a2a2e39d2798bd3afbd6fa023b56fa493f8f21d5e9507cd6b" gracePeriod=25
Oct 29 23:41:03.586768 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:03.586728    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:03.734437 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:03.734409    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w"
Oct 29 23:41:03.799076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:03.799039    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c784cb9f-4716-4092-bf8a-5b7637a23a0e-kube-api-access-llnfs\" (UniqueName: \"kubernetes.io/projected/c784cb9f-4716-4092-bf8a-5b7637a23a0e-kube-api-access-llnfs\") pod \"c784cb9f-4716-4092-bf8a-5b7637a23a0e\" (UID: \"c784cb9f-4716-4092-bf8a-5b7637a23a0e\") "
Oct 29 23:41:03.799445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:03.799422    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c784cb9f-4716-4092-bf8a-5b7637a23a0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c784cb9f-4716-4092-bf8a-5b7637a23a0e-plugins-config-volume\") pod \"c784cb9f-4716-4092-bf8a-5b7637a23a0e\" (UID: \"c784cb9f-4716-4092-bf8a-5b7637a23a0e\") "
Oct 29 23:41:03.799915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:03.799885    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c784cb9f-4716-4092-bf8a-5b7637a23a0e-plugins-config-volume" pod "c784cb9f-4716-4092-bf8a-5b7637a23a0e" (UID: "c784cb9f-4716-4092-bf8a-5b7637a23a0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:41:03.800230 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:03.800201    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c784cb9f-4716-4092-bf8a-5b7637a23a0e-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:41:03.801358 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:03.801331    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c784cb9f-4716-4092-bf8a-5b7637a23a0e-kube-api-access-llnfs" pod "c784cb9f-4716-4092-bf8a-5b7637a23a0e" (UID: "c784cb9f-4716-4092-bf8a-5b7637a23a0e"). InnerVolumeSpecName "kube-api-access-llnfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:41:03.901716 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:03.901686    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llnfs\" (UniqueName: \"kubernetes.io/projected/c784cb9f-4716-4092-bf8a-5b7637a23a0e-kube-api-access-llnfs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:41:04.235059 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:04.235022    5428 generic.go:356] "Generic (PLEG): container finished" podID="c784cb9f-4716-4092-bf8a-5b7637a23a0e" containerID="8d0e67deded15c7a2a2e39d2798bd3afbd6fa023b56fa493f8f21d5e9507cd6b" exitCode=0
Oct 29 23:41:04.235234 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:04.235067    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w"
Oct 29 23:41:04.235234 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:04.235098    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w" event={"ID":"c784cb9f-4716-4092-bf8a-5b7637a23a0e","Type":"ContainerDied","Data":"8d0e67deded15c7a2a2e39d2798bd3afbd6fa023b56fa493f8f21d5e9507cd6b"}
Oct 29 23:41:04.235234 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:04.235149    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w" event={"ID":"c784cb9f-4716-4092-bf8a-5b7637a23a0e","Type":"ContainerDied","Data":"9ffde4de7c8f4d97f1518a11d33e284504b8d3c04587914541fe6be28a814fe6"}
Oct 29 23:41:04.235234 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:04.235181    5428 scope.go:122] "RemoveContainer" containerID="8d0e67deded15c7a2a2e39d2798bd3afbd6fa023b56fa493f8f21d5e9507cd6b"
Oct 29 23:41:04.248022 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:04.248004    5428 scope.go:122] "RemoveContainer" containerID="8d0e67deded15c7a2a2e39d2798bd3afbd6fa023b56fa493f8f21d5e9507cd6b"
Oct 29 23:41:04.248276 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:41:04.248256    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0e67deded15c7a2a2e39d2798bd3afbd6fa023b56fa493f8f21d5e9507cd6b\": container with ID starting with 8d0e67deded15c7a2a2e39d2798bd3afbd6fa023b56fa493f8f21d5e9507cd6b not found: ID does not exist" containerID="8d0e67deded15c7a2a2e39d2798bd3afbd6fa023b56fa493f8f21d5e9507cd6b"
Oct 29 23:41:04.248357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:04.248282    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0e67deded15c7a2a2e39d2798bd3afbd6fa023b56fa493f8f21d5e9507cd6b"} err="failed to get container status \"8d0e67deded15c7a2a2e39d2798bd3afbd6fa023b56fa493f8f21d5e9507cd6b\": rpc error: code = NotFound desc = could not find container \"8d0e67deded15c7a2a2e39d2798bd3afbd6fa023b56fa493f8f21d5e9507cd6b\": container with ID starting with 8d0e67deded15c7a2a2e39d2798bd3afbd6fa023b56fa493f8f21d5e9507cd6b not found: ID does not exist"
Oct 29 23:41:04.289491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:04.289458    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w"]
Oct 29 23:41:04.301276 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:04.301245    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9965bf-659ccbcfb-9lp8w"]
Oct 29 23:41:04.780881 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:04.780850    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c784cb9f-4716-4092-bf8a-5b7637a23a0e" path="/var/lib/kubelet/pods/c784cb9f-4716-4092-bf8a-5b7637a23a0e/volumes"
Oct 29 23:41:05.409110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:05.409076    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:06.544918 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:06.544886    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:06.668234 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:06.668198    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:07.397291 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:07.397259    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:08.284473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:08.284440    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:08.400595 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:08.400564    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:08.562736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:08.562659    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8"]
Oct 29 23:41:08.563841 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:08.563779    5428 cpu_manager.go:414] "containerMap: removing container" podUID="c784cb9f-4716-4092-bf8a-5b7637a23a0e" containerName="payload-processing"
Oct 29 23:41:08.563841 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:08.563817    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c784cb9f-4716-4092-bf8a-5b7637a23a0e" containerName="payload-processing"
Oct 29 23:41:08.564081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:08.564065    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c784cb9f-4716-4092-bf8a-5b7637a23a0e" containerName="payload-processing"
Oct 29 23:41:08.574232 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:08.574188    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8"
Oct 29 23:41:08.630357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:08.609172    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d20345-dockercfg-zf6ql\""
Oct 29 23:41:08.630357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:08.609279    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d20345\""
Oct 29 23:41:08.687474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:08.686868    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f792ad2-06af-45dc-92a3-3e5514e9201a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8\" (UID: \"3f792ad2-06af-45dc-92a3-3e5514e9201a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8"
Oct 29 23:41:08.687474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:08.687215    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgjjd\" (UniqueName: \"kubernetes.io/projected/3f792ad2-06af-45dc-92a3-3e5514e9201a-kube-api-access-mgjjd\") pod \"payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8\" (UID: \"3f792ad2-06af-45dc-92a3-3e5514e9201a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8"
Oct 29 23:41:08.738993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:08.738959    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8"]
Oct 29 23:41:08.788190 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:08.788153    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f792ad2-06af-45dc-92a3-3e5514e9201a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8\" (UID: \"3f792ad2-06af-45dc-92a3-3e5514e9201a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8"
Oct 29 23:41:08.788363 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:08.788336    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgjjd\" (UniqueName: \"kubernetes.io/projected/3f792ad2-06af-45dc-92a3-3e5514e9201a-kube-api-access-mgjjd\") pod \"payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8\" (UID: \"3f792ad2-06af-45dc-92a3-3e5514e9201a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8"
Oct 29 23:41:08.788981 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:08.788963    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f792ad2-06af-45dc-92a3-3e5514e9201a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8\" (UID: \"3f792ad2-06af-45dc-92a3-3e5514e9201a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8"
Oct 29 23:41:08.812076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:08.812048    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgjjd\" (UniqueName: \"kubernetes.io/projected/3f792ad2-06af-45dc-92a3-3e5514e9201a-kube-api-access-mgjjd\") pod \"payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8\" (UID: \"3f792ad2-06af-45dc-92a3-3e5514e9201a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8"
Oct 29 23:41:09.021125 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:09.021095    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8"
Oct 29 23:41:09.336321 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:09.336292    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8"]
Oct 29 23:41:09.336705 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:41:09.336677    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f792ad2_06af_45dc_92a3_3e5514e9201a.slice/crio-0ede949a20f997ad8466a11535ce8e01266a612205e7a2a3f52a9869ade2736d WatchSource:0}: Error finding container 0ede949a20f997ad8466a11535ce8e01266a612205e7a2a3f52a9869ade2736d: Status 404 returned error can't find the container with id 0ede949a20f997ad8466a11535ce8e01266a612205e7a2a3f52a9869ade2736d
Oct 29 23:41:09.585046 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:09.584968    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:10.298292 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:10.298249    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8" event={"ID":"3f792ad2-06af-45dc-92a3-3e5514e9201a","Type":"ContainerStarted","Data":"10e72c50b321c754dc6d460b78dab117d96e70792f0567a0b7954f3703e56a4e"}
Oct 29 23:41:10.298469 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:10.298317    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8" event={"ID":"3f792ad2-06af-45dc-92a3-3e5514e9201a","Type":"ContainerStarted","Data":"0ede949a20f997ad8466a11535ce8e01266a612205e7a2a3f52a9869ade2736d"}
Oct 29 23:41:10.298469 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:10.298351    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8"
Oct 29 23:41:10.354089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:10.353967    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8" podStartSLOduration=2.353946352 podStartE2EDuration="2.353946352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:41:10.342347894 +0000 UTC m=+3473.685296640" watchObservedRunningTime="2026-10-29 23:41:10.353946352 +0000 UTC m=+3473.696895105"
Oct 29 23:41:11.393848 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:11.393797    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:13.003786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:13.003748    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:14.679520 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:14.679482    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:16.354064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:16.354034    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:17.370344 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:41:17.370303    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:41:18.077433 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:18.077400    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:19.690319 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:19.690288    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:21.310704 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:21.310666    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8"
Oct 29 23:41:21.484830 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:21.484787    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:23.078339 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:23.078306    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:24.783733 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:24.783696    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:26.456402 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:26.456369    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:28.275420 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:28.275389    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:29.986614 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:29.986577    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:31.610874 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:31.610836    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:32.385462 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:32.385428    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:33.151860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:33.151824    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:33.267280 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:33.267248    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:34.938710 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:34.938659    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:35.066439 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:35.066310    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:35.320126 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:35.320042    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:36.712193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:36.712159    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:38.378416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:38.378379    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:40.076397 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:40.076364    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:41.851605 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:41.851572    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:43.513239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:43.513208    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:45.281368 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:45.281334    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:46.877697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:46.877662    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:47.371357 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:41:47.371318    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:41:49.049169 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:49.049140    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:51.021892 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:51.021860    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:52.981175 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:52.981142    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:54.747720 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:54.747682    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:56.559238 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:56.559208    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:58.241772 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:58.241691    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:41:59.970911 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:41:59.970873    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:01.210788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:01.210758    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:01.697267 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:01.697236    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:01.820378 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:01.820344    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:02.207217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:02.207179    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:02.955028 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:02.954989    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:04.206729 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:04.206688    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:06.023739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:06.023680    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:07.470384 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:07.470349    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"]
Oct 29 23:42:07.470925 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:07.470571    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296" podUID="0886ba54-dbe9-4839-af08-89e8c5b47422" containerName="maas-api" containerID="cri-o://7deb798381d28e9480ae3982c47d369fe542196eaa8cd1bba7ec034b404d9835" gracePeriod=30
Oct 29 23:42:07.556729 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:07.556692    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:07.833369 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:07.833339    5428 generic.go:356] "Generic (PLEG): container finished" podID="0886ba54-dbe9-4839-af08-89e8c5b47422" containerID="7deb798381d28e9480ae3982c47d369fe542196eaa8cd1bba7ec034b404d9835" exitCode=0
Oct 29 23:42:07.833507 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:07.833408    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296" event={"ID":"0886ba54-dbe9-4839-af08-89e8c5b47422","Type":"ContainerDied","Data":"7deb798381d28e9480ae3982c47d369fe542196eaa8cd1bba7ec034b404d9835"}
Oct 29 23:42:07.833507 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:07.833463    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296" event={"ID":"0886ba54-dbe9-4839-af08-89e8c5b47422","Type":"ContainerDied","Data":"b439772dbd1b1b115e3d7901e16f992688b314acd23cfe1a5bdf5ffb162f3a80"}
Oct 29 23:42:07.833507 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:07.833479    5428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b439772dbd1b1b115e3d7901e16f992688b314acd23cfe1a5bdf5ffb162f3a80"
Oct 29 23:42:07.840613 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:07.840587    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"
Oct 29 23:42:07.906822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:07.900532    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"]
Oct 29 23:42:07.906822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:07.902238    5428 cpu_manager.go:414] "containerMap: removing container" podUID="0886ba54-dbe9-4839-af08-89e8c5b47422" containerName="maas-api"
Oct 29 23:42:07.906822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:07.902272    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0886ba54-dbe9-4839-af08-89e8c5b47422" containerName="maas-api"
Oct 29 23:42:07.906822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:07.902549    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="0886ba54-dbe9-4839-af08-89e8c5b47422" containerName="maas-api"
Oct 29 23:42:07.914862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:07.907907    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:07.926954 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:07.924074    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-825472-openshift-default-dockercfg-vh6rb\""
Oct 29 23:42:08.017289 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:07.999226    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0886ba54-dbe9-4839-af08-89e8c5b47422-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0886ba54-dbe9-4839-af08-89e8c5b47422-openshift-service-ca\") pod \"0886ba54-dbe9-4839-af08-89e8c5b47422\" (UID: \"0886ba54-dbe9-4839-af08-89e8c5b47422\") "
Oct 29 23:42:08.017289 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:07.999339    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0886ba54-dbe9-4839-af08-89e8c5b47422-kube-api-access-hz5n6\" (UniqueName: \"kubernetes.io/projected/0886ba54-dbe9-4839-af08-89e8c5b47422-kube-api-access-hz5n6\") pod \"0886ba54-dbe9-4839-af08-89e8c5b47422\" (UID: \"0886ba54-dbe9-4839-af08-89e8c5b47422\") "
Oct 29 23:42:08.017289 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:07.999448    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0886ba54-dbe9-4839-af08-89e8c5b47422-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0886ba54-dbe9-4839-af08-89e8c5b47422-maas-api-tls\") pod \"0886ba54-dbe9-4839-af08-89e8c5b47422\" (UID: \"0886ba54-dbe9-4839-af08-89e8c5b47422\") "
Oct 29 23:42:08.017289 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:07.999682    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0886ba54-dbe9-4839-af08-89e8c5b47422-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0886ba54-dbe9-4839-af08-89e8c5b47422-metrics-certs\") pod \"0886ba54-dbe9-4839-af08-89e8c5b47422\" (UID: \"0886ba54-dbe9-4839-af08-89e8c5b47422\") "
Oct 29 23:42:08.024823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.017909    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0886ba54-dbe9-4839-af08-89e8c5b47422-openshift-service-ca" pod "0886ba54-dbe9-4839-af08-89e8c5b47422" (UID: "0886ba54-dbe9-4839-af08-89e8c5b47422"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:42:08.024823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.020724    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0886ba54-dbe9-4839-af08-89e8c5b47422-maas-api-tls" pod "0886ba54-dbe9-4839-af08-89e8c5b47422" (UID: "0886ba54-dbe9-4839-af08-89e8c5b47422"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:42:08.108919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.104157    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.108919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.104281    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.108919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.104321    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772a13d1-e842-4cf0-b36e-09e07d7936e4-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.108919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.104370    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf9lw\" (UniqueName: \"kubernetes.io/projected/772a13d1-e842-4cf0-b36e-09e07d7936e4-kube-api-access-qf9lw\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.108919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.104552    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.108919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.104651    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.108919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.104703    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.108919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.104740    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.108919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.104790    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.108919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.104971    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0886ba54-dbe9-4839-af08-89e8c5b47422-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:42:08.108919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.104989    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0886ba54-dbe9-4839-af08-89e8c5b47422-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:42:08.108919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.105174    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"]
Oct 29 23:42:08.110977 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.110942    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0886ba54-dbe9-4839-af08-89e8c5b47422-kube-api-access-hz5n6" pod "0886ba54-dbe9-4839-af08-89e8c5b47422" (UID: "0886ba54-dbe9-4839-af08-89e8c5b47422"). InnerVolumeSpecName "kube-api-access-hz5n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:42:08.114991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.112983    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0886ba54-dbe9-4839-af08-89e8c5b47422-metrics-certs" pod "0886ba54-dbe9-4839-af08-89e8c5b47422" (UID: "0886ba54-dbe9-4839-af08-89e8c5b47422"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:42:08.206547 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.206509    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.206734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.206559    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.206734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.206586    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.206883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.206762    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.206964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.206937    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.207055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.207040    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.207117 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.207054    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.207117 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.207073    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772a13d1-e842-4cf0-b36e-09e07d7936e4-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.207117 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.207088    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.207290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.207147    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf9lw\" (UniqueName: \"kubernetes.io/projected/772a13d1-e842-4cf0-b36e-09e07d7936e4-kube-api-access-qf9lw\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.207290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.207277    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.207428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.207396    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.207517 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.207502    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0886ba54-dbe9-4839-af08-89e8c5b47422-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:42:08.207584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.207525    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz5n6\" (UniqueName: \"kubernetes.io/projected/0886ba54-dbe9-4839-af08-89e8c5b47422-kube-api-access-hz5n6\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:42:08.207731 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.207707    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.207822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.207752    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772a13d1-e842-4cf0-b36e-09e07d7936e4-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.209148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.209125    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.209585 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.209561    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.230648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.230618    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.230872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.230853    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf9lw\" (UniqueName: \"kubernetes.io/projected/772a13d1-e842-4cf0-b36e-09e07d7936e4-kube-api-access-qf9lw\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.336867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.336797    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:08.358913 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.358875    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:08.591480 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.591450    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:08.697900 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.697860    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:08.713476 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.713447    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"]
Oct 29 23:42:08.713657 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:42:08.713630    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772a13d1_e842_4cf0_b36e_09e07d7936e4.slice/crio-070f1655c4cec3cba1e1e3b2006fe1cd0774113dd9bb4e9fdf76710172bbbbcd WatchSource:0}: Error finding container 070f1655c4cec3cba1e1e3b2006fe1cd0774113dd9bb4e9fdf76710172bbbbcd: Status 404 returned error can't find the container with id 070f1655c4cec3cba1e1e3b2006fe1cd0774113dd9bb4e9fdf76710172bbbbcd
Oct 29 23:42:08.717282 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.717242    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:42:08.717383 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.717331    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:42:08.717383 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.717376    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:42:08.845023 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.844996    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2" event={"ID":"772a13d1-e842-4cf0-b36e-09e07d7936e4","Type":"ContainerStarted","Data":"070f1655c4cec3cba1e1e3b2006fe1cd0774113dd9bb4e9fdf76710172bbbbcd"}
Oct 29 23:42:08.845179 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.845094    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"
Oct 29 23:42:08.903056 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.902459    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"]
Oct 29 23:42:08.907656 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:08.907622    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9965bf-584c4c47f4-vm296"]
Oct 29 23:42:09.127462 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:09.127433    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:09.855071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:09.855035    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2" event={"ID":"772a13d1-e842-4cf0-b36e-09e07d7936e4","Type":"ContainerStarted","Data":"aee97831fe33e6a456d2c3de5ad3b8878f14f57135fe327039c1d05ee651d359"}
Oct 29 23:42:09.928039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:09.927989    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2" podStartSLOduration=102.927974608 podStartE2EDuration="1m42.927974608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:42:09.913047857 +0000 UTC m=+3533.255996603" watchObservedRunningTime="2026-10-29 23:42:09.927974608 +0000 UTC m=+3533.270923352"
Oct 29 23:42:10.359226 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:10.359194    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:10.367681 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:10.367653    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:10.614975 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:10.614938    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:10.781302 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:10.781267    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0886ba54-dbe9-4839-af08-89e8c5b47422" path="/var/lib/kubelet/pods/0886ba54-dbe9-4839-af08-89e8c5b47422/volumes"
Oct 29 23:42:10.864470 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:10.864442    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:10.865737 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:10.865680    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:42:12.428594 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:12.428550    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:12.553176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:12.553142    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:14.581983 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:14.581943    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:14.697197 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:14.697163    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:14.731208 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:14.731178    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz"]
Oct 29 23:42:14.737803 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:14.737784    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz"
Oct 29 23:42:14.760953 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:14.760917    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz"]
Oct 29 23:42:14.770186 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:14.770139    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-825472-dockercfg-mt4lv\""
Oct 29 23:42:14.828705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:14.828668    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13040ff3-bf4c-4e93-8823-1feb043a8eda-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz\" (UID: \"13040ff3-bf4c-4e93-8823-1feb043a8eda\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz"
Oct 29 23:42:14.829093 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:14.829062    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8xc5\" (UniqueName: \"kubernetes.io/projected/13040ff3-bf4c-4e93-8823-1feb043a8eda-kube-api-access-l8xc5\") pod \"payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz\" (UID: \"13040ff3-bf4c-4e93-8823-1feb043a8eda\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz"
Oct 29 23:42:14.834908 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:14.834843    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-825472\""
Oct 29 23:42:14.945600 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:14.945568    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13040ff3-bf4c-4e93-8823-1feb043a8eda-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz\" (UID: \"13040ff3-bf4c-4e93-8823-1feb043a8eda\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz"
Oct 29 23:42:14.946045 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:14.946019    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8xc5\" (UniqueName: \"kubernetes.io/projected/13040ff3-bf4c-4e93-8823-1feb043a8eda-kube-api-access-l8xc5\") pod \"payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz\" (UID: \"13040ff3-bf4c-4e93-8823-1feb043a8eda\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz"
Oct 29 23:42:14.951897 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:14.951873    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13040ff3-bf4c-4e93-8823-1feb043a8eda-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz\" (UID: \"13040ff3-bf4c-4e93-8823-1feb043a8eda\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz"
Oct 29 23:42:14.971680 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:14.971648    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8xc5\" (UniqueName: \"kubernetes.io/projected/13040ff3-bf4c-4e93-8823-1feb043a8eda-kube-api-access-l8xc5\") pod \"payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz\" (UID: \"13040ff3-bf4c-4e93-8823-1feb043a8eda\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz"
Oct 29 23:42:15.056649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:15.056616    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz"
Oct 29 23:42:15.382688 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:15.382657    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz"]
Oct 29 23:42:15.383158 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:42:15.383131    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13040ff3_bf4c_4e93_8823_1feb043a8eda.slice/crio-25aef15166d097474154c05c7d189bec9daa668444c1e84836cba516fc939567 WatchSource:0}: Error finding container 25aef15166d097474154c05c7d189bec9daa668444c1e84836cba516fc939567: Status 404 returned error can't find the container with id 25aef15166d097474154c05c7d189bec9daa668444c1e84836cba516fc939567
Oct 29 23:42:15.781926 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:15.781844    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:15.900766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:15.900734    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:15.953229 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:15.953192    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz" event={"ID":"13040ff3-bf4c-4e93-8823-1feb043a8eda","Type":"ContainerStarted","Data":"9e48724303f4691e8d44e8a6e6a2537247c8bf859fab16e749e11bf72b1eaffd"}
Oct 29 23:42:15.953439 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:15.953249    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz" event={"ID":"13040ff3-bf4c-4e93-8823-1feb043a8eda","Type":"ContainerStarted","Data":"25aef15166d097474154c05c7d189bec9daa668444c1e84836cba516fc939567"}
Oct 29 23:42:15.953439 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:15.953327    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz"
Oct 29 23:42:15.994441 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:15.994379    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz" podStartSLOduration=1.994362698 podStartE2EDuration="1.994362698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:42:15.993188481 +0000 UTC m=+3539.336137228" watchObservedRunningTime="2026-10-29 23:42:15.994362698 +0000 UTC m=+3539.337311435"
Oct 29 23:42:16.141781 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:16.141743    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:16.993872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:16.993840    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:17.373338 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:42:17.373300    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:42:18.825597 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:18.823148    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:20.496567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:20.496523    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:22.264113 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:22.264077    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:23.927110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:23.927060    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:24.052753 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:24.052717    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:24.091012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:24.090978    5428 scope.go:122] "RemoveContainer" containerID="7deb798381d28e9480ae3982c47d369fe542196eaa8cd1bba7ec034b404d9835"
Oct 29 23:42:25.885829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:25.885777    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:26.965078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:26.965044    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz"
Oct 29 23:42:27.431639 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:27.431593    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq"]
Oct 29 23:42:27.464539 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:27.464506    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv"]
Oct 29 23:42:27.570841 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:27.569246    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:27.681129 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:27.681074    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:29.469776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:29.469738    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:31.172101 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:31.172070    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:32.432306 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:32.432270    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq" podUID="5186bf82-c9e1-49e3-9dd0-f362ddce36ce" containerName="payload-processing" containerID="cri-o://86e21cfb98f5cf2370c3a3c36fdc7fe6f72183b6ed4e332e36f6d9712ed929cf" gracePeriod=25
Oct 29 23:42:32.465047 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:32.465013    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv" podUID="8d42d182-5615-4c04-a4af-5d7aee1c4dba" containerName="payload-pre-processing" containerID="cri-o://490e845a407834d6ff17eaa047e1e2ac5a2d46f8bbddbaf0baa4db57ca290642" gracePeriod=25
Oct 29 23:42:32.544675 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:32.544635    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:32.907508 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:32.907460    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:33.278414 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:33.278339    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:34.182758 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:34.182716    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv" podUID="8d42d182-5615-4c04-a4af-5d7aee1c4dba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:42:34.183191 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:34.182775    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq" podUID="5186bf82-c9e1-49e3-9dd0-f362ddce36ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:42:34.571728 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:34.571615    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:36.200152 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:36.200091    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:37.882085 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:37.882047    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:39.668760 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:39.668726    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:41.378778 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:41.378726    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:43.108578 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:43.108542    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:44.183419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:44.183383    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq" podUID="5186bf82-c9e1-49e3-9dd0-f362ddce36ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:42:44.183853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:44.183453    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv" podUID="8d42d182-5615-4c04-a4af-5d7aee1c4dba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:42:45.366739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:45.366698    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:47.160618 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:47.160582    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:47.370186 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:42:47.370147    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:42:48.996322 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:48.996281    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:50.645593 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:50.645559    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:52.354763 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:52.354729    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:52.805336 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:52.805305    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq"
Oct 29 23:42:52.809606 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:52.809585    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv"
Oct 29 23:42:52.891625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:52.891530    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d42d182-5615-4c04-a4af-5d7aee1c4dba-kube-api-access-jv572\" (UniqueName: \"kubernetes.io/projected/8d42d182-5615-4c04-a4af-5d7aee1c4dba-kube-api-access-jv572\") pod \"8d42d182-5615-4c04-a4af-5d7aee1c4dba\" (UID: \"8d42d182-5615-4c04-a4af-5d7aee1c4dba\") "
Oct 29 23:42:52.891818 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:52.891635    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d42d182-5615-4c04-a4af-5d7aee1c4dba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d42d182-5615-4c04-a4af-5d7aee1c4dba-plugins-config-volume\") pod \"8d42d182-5615-4c04-a4af-5d7aee1c4dba\" (UID: \"8d42d182-5615-4c04-a4af-5d7aee1c4dba\") "
Oct 29 23:42:52.891818 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:52.891752    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5186bf82-c9e1-49e3-9dd0-f362ddce36ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5186bf82-c9e1-49e3-9dd0-f362ddce36ce-plugins-config-volume\") pod \"5186bf82-c9e1-49e3-9dd0-f362ddce36ce\" (UID: \"5186bf82-c9e1-49e3-9dd0-f362ddce36ce\") "
Oct 29 23:42:52.891818 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:52.891781    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5186bf82-c9e1-49e3-9dd0-f362ddce36ce-kube-api-access-tdw5f\" (UniqueName: \"kubernetes.io/projected/5186bf82-c9e1-49e3-9dd0-f362ddce36ce-kube-api-access-tdw5f\") pod \"5186bf82-c9e1-49e3-9dd0-f362ddce36ce\" (UID: \"5186bf82-c9e1-49e3-9dd0-f362ddce36ce\") "
Oct 29 23:42:52.892356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:52.892313    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d42d182-5615-4c04-a4af-5d7aee1c4dba-plugins-config-volume" pod "8d42d182-5615-4c04-a4af-5d7aee1c4dba" (UID: "8d42d182-5615-4c04-a4af-5d7aee1c4dba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:42:52.893480 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:52.892995    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5186bf82-c9e1-49e3-9dd0-f362ddce36ce-plugins-config-volume" pod "5186bf82-c9e1-49e3-9dd0-f362ddce36ce" (UID: "5186bf82-c9e1-49e3-9dd0-f362ddce36ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:42:52.893480 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:52.893224    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d42d182-5615-4c04-a4af-5d7aee1c4dba-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:42:52.893480 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:52.893247    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5186bf82-c9e1-49e3-9dd0-f362ddce36ce-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:42:52.895211 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:52.895179    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d42d182-5615-4c04-a4af-5d7aee1c4dba-kube-api-access-jv572" pod "8d42d182-5615-4c04-a4af-5d7aee1c4dba" (UID: "8d42d182-5615-4c04-a4af-5d7aee1c4dba"). InnerVolumeSpecName "kube-api-access-jv572". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:42:52.896812 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:52.896781    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5186bf82-c9e1-49e3-9dd0-f362ddce36ce-kube-api-access-tdw5f" pod "5186bf82-c9e1-49e3-9dd0-f362ddce36ce" (UID: "5186bf82-c9e1-49e3-9dd0-f362ddce36ce"). InnerVolumeSpecName "kube-api-access-tdw5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:42:52.994822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:52.994773    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv572\" (UniqueName: \"kubernetes.io/projected/8d42d182-5615-4c04-a4af-5d7aee1c4dba-kube-api-access-jv572\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:42:52.994974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:52.994830    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdw5f\" (UniqueName: \"kubernetes.io/projected/5186bf82-c9e1-49e3-9dd0-f362ddce36ce-kube-api-access-tdw5f\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:42:53.387463 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:53.387431    5428 generic.go:356] "Generic (PLEG): container finished" podID="5186bf82-c9e1-49e3-9dd0-f362ddce36ce" containerID="86e21cfb98f5cf2370c3a3c36fdc7fe6f72183b6ed4e332e36f6d9712ed929cf" exitCode=0
Oct 29 23:42:53.388016 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:53.387488    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq"
Oct 29 23:42:53.388016 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:53.387513    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq" event={"ID":"5186bf82-c9e1-49e3-9dd0-f362ddce36ce","Type":"ContainerDied","Data":"86e21cfb98f5cf2370c3a3c36fdc7fe6f72183b6ed4e332e36f6d9712ed929cf"}
Oct 29 23:42:53.388016 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:53.387566    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq" event={"ID":"5186bf82-c9e1-49e3-9dd0-f362ddce36ce","Type":"ContainerDied","Data":"14f55f69dd3864c0e63d2eaa5298ab0e9cf9fdf277e213bf2354f047d3c04c5e"}
Oct 29 23:42:53.388016 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:53.387590    5428 scope.go:122] "RemoveContainer" containerID="86e21cfb98f5cf2370c3a3c36fdc7fe6f72183b6ed4e332e36f6d9712ed929cf"
Oct 29 23:42:53.389247 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:53.389225    5428 generic.go:356] "Generic (PLEG): container finished" podID="8d42d182-5615-4c04-a4af-5d7aee1c4dba" containerID="490e845a407834d6ff17eaa047e1e2ac5a2d46f8bbddbaf0baa4db57ca290642" exitCode=0
Oct 29 23:42:53.389384 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:53.389267    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv"
Oct 29 23:42:53.389384 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:53.389289    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv" event={"ID":"8d42d182-5615-4c04-a4af-5d7aee1c4dba","Type":"ContainerDied","Data":"490e845a407834d6ff17eaa047e1e2ac5a2d46f8bbddbaf0baa4db57ca290642"}
Oct 29 23:42:53.389384 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:53.389325    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv" event={"ID":"8d42d182-5615-4c04-a4af-5d7aee1c4dba","Type":"ContainerDied","Data":"c0ecf0eb7e7cdfa16d2da8be80cc7ed91e59480fb66016d90efffee17d0d30e9"}
Oct 29 23:42:53.400243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:53.400202    5428 scope.go:122] "RemoveContainer" containerID="86e21cfb98f5cf2370c3a3c36fdc7fe6f72183b6ed4e332e36f6d9712ed929cf"
Oct 29 23:42:53.400500 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:42:53.400475    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e21cfb98f5cf2370c3a3c36fdc7fe6f72183b6ed4e332e36f6d9712ed929cf\": container with ID starting with 86e21cfb98f5cf2370c3a3c36fdc7fe6f72183b6ed4e332e36f6d9712ed929cf not found: ID does not exist" containerID="86e21cfb98f5cf2370c3a3c36fdc7fe6f72183b6ed4e332e36f6d9712ed929cf"
Oct 29 23:42:53.400610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:53.400506    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e21cfb98f5cf2370c3a3c36fdc7fe6f72183b6ed4e332e36f6d9712ed929cf"} err="failed to get container status \"86e21cfb98f5cf2370c3a3c36fdc7fe6f72183b6ed4e332e36f6d9712ed929cf\": rpc error: code = NotFound desc = could not find container \"86e21cfb98f5cf2370c3a3c36fdc7fe6f72183b6ed4e332e36f6d9712ed929cf\": container with ID starting with 86e21cfb98f5cf2370c3a3c36fdc7fe6f72183b6ed4e332e36f6d9712ed929cf not found: ID does not exist"
Oct 29 23:42:53.400610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:53.400522    5428 scope.go:122] "RemoveContainer" containerID="490e845a407834d6ff17eaa047e1e2ac5a2d46f8bbddbaf0baa4db57ca290642"
Oct 29 23:42:53.411700 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:53.411676    5428 scope.go:122] "RemoveContainer" containerID="490e845a407834d6ff17eaa047e1e2ac5a2d46f8bbddbaf0baa4db57ca290642"
Oct 29 23:42:53.412113 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:42:53.412080    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490e845a407834d6ff17eaa047e1e2ac5a2d46f8bbddbaf0baa4db57ca290642\": container with ID starting with 490e845a407834d6ff17eaa047e1e2ac5a2d46f8bbddbaf0baa4db57ca290642 not found: ID does not exist" containerID="490e845a407834d6ff17eaa047e1e2ac5a2d46f8bbddbaf0baa4db57ca290642"
Oct 29 23:42:53.412224 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:53.412119    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490e845a407834d6ff17eaa047e1e2ac5a2d46f8bbddbaf0baa4db57ca290642"} err="failed to get container status \"490e845a407834d6ff17eaa047e1e2ac5a2d46f8bbddbaf0baa4db57ca290642\": rpc error: code = NotFound desc = could not find container \"490e845a407834d6ff17eaa047e1e2ac5a2d46f8bbddbaf0baa4db57ca290642\": container with ID starting with 490e845a407834d6ff17eaa047e1e2ac5a2d46f8bbddbaf0baa4db57ca290642 not found: ID does not exist"
Oct 29 23:42:53.441486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:53.441454    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv"]
Oct 29 23:42:53.452677 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:53.452653    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c5ef2f-86c8676f5-jqhnv"]
Oct 29 23:42:53.486242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:53.486215    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq"]
Oct 29 23:42:53.491074 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:53.491048    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c5ef2f-5d4d9d5d-9fpvq"]
Oct 29 23:42:54.163985 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:54.163955    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:54.780416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:54.780378    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5186bf82-c9e1-49e3-9dd0-f362ddce36ce" path="/var/lib/kubelet/pods/5186bf82-c9e1-49e3-9dd0-f362ddce36ce/volumes"
Oct 29 23:42:54.780948 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:54.780924    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d42d182-5615-4c04-a4af-5d7aee1c4dba" path="/var/lib/kubelet/pods/8d42d182-5615-4c04-a4af-5d7aee1c4dba/volumes"
Oct 29 23:42:55.776766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:55.776732    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:57.439106 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:57.439073    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:58.835505 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:58.835444    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:59.700241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:59.700207    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:59.822288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:59.822246    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:42:59.948033 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:42:59.947986    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:00.375856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:00.375810    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:01.209791 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:01.209759    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:01.956530 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:01.956492    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:03.678531 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:03.678491    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:05.440861 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:05.440824    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:06.263844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.263782    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"]
Oct 29 23:43:06.264229 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.264205    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46" podUID="43df95b6-fe67-4edd-aeea-ff29d29c94c2" containerName="discovery" containerID="cri-o://8d086784ef382515f0d9a163d5af9cf9528550b9e7f2f1e84b1b051af6c44f0d" gracePeriod=30
Oct 29 23:43:06.528912 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.528787    5428 generic.go:356] "Generic (PLEG): container finished" podID="43df95b6-fe67-4edd-aeea-ff29d29c94c2" containerID="8d086784ef382515f0d9a163d5af9cf9528550b9e7f2f1e84b1b051af6c44f0d" exitCode=0
Oct 29 23:43:06.528912 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.528852    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46" event={"ID":"43df95b6-fe67-4edd-aeea-ff29d29c94c2","Type":"ContainerDied","Data":"8d086784ef382515f0d9a163d5af9cf9528550b9e7f2f1e84b1b051af6c44f0d"}
Oct 29 23:43:06.582581 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.582549    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:43:06.630432 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.630382    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-kubeconfig\") pod \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") "
Oct 29 23:43:06.641988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.630939    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43df95b6-fe67-4edd-aeea-ff29d29c94c2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/43df95b6-fe67-4edd-aeea-ff29d29c94c2-local-certs\") pod \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") "
Oct 29 23:43:06.641988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.633539    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43df95b6-fe67-4edd-aeea-ff29d29c94c2-local-certs" pod "43df95b6-fe67-4edd-aeea-ff29d29c94c2" (UID: "43df95b6-fe67-4edd-aeea-ff29d29c94c2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:43:06.641988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.633710    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-kubeconfig" pod "43df95b6-fe67-4edd-aeea-ff29d29c94c2" (UID: "43df95b6-fe67-4edd-aeea-ff29d29c94c2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:43:06.641988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.634286    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-token" pod "43df95b6-fe67-4edd-aeea-ff29d29c94c2" (UID: "43df95b6-fe67-4edd-aeea-ff29d29c94c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:43:06.641988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.638737    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-token\" (UniqueName: \"kubernetes.io/projected/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-token\") pod \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") "
Oct 29 23:43:06.641988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.638875    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-cacerts\" (UniqueName: \"kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-cacerts\") pod \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") "
Oct 29 23:43:06.641988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.638947    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-csr-ca-configmap\") pod \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") "
Oct 29 23:43:06.641988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.639060    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43df95b6-fe67-4edd-aeea-ff29d29c94c2-kube-api-access-zh66c\" (UniqueName: \"kubernetes.io/projected/43df95b6-fe67-4edd-aeea-ff29d29c94c2-kube-api-access-zh66c\") pod \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") "
Oct 29 23:43:06.641988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.639115    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-csr-dns-cert\") pod \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\" (UID: \"43df95b6-fe67-4edd-aeea-ff29d29c94c2\") "
Oct 29 23:43:06.641988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.639495    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-csr-ca-configmap" pod "43df95b6-fe67-4edd-aeea-ff29d29c94c2" (UID: "43df95b6-fe67-4edd-aeea-ff29d29c94c2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:43:06.641988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.639976    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:06.641988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.640000    5428 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-csr-ca-configmap\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:06.641988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.640014    5428 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-kubeconfig\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:06.641988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.640027    5428 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/43df95b6-fe67-4edd-aeea-ff29d29c94c2-local-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:06.644855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.644820    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-csr-dns-cert" pod "43df95b6-fe67-4edd-aeea-ff29d29c94c2" (UID: "43df95b6-fe67-4edd-aeea-ff29d29c94c2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:43:06.647241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.647204    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43df95b6-fe67-4edd-aeea-ff29d29c94c2-kube-api-access-zh66c" pod "43df95b6-fe67-4edd-aeea-ff29d29c94c2" (UID: "43df95b6-fe67-4edd-aeea-ff29d29c94c2"). InnerVolumeSpecName "kube-api-access-zh66c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:43:06.647356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.647301    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-cacerts" pod "43df95b6-fe67-4edd-aeea-ff29d29c94c2" (UID: "43df95b6-fe67-4edd-aeea-ff29d29c94c2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:43:06.653424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.651023    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"]
Oct 29 23:43:06.653424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.652534    5428 cpu_manager.go:414] "containerMap: removing container" podUID="43df95b6-fe67-4edd-aeea-ff29d29c94c2" containerName="discovery"
Oct 29 23:43:06.653424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.652552    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43df95b6-fe67-4edd-aeea-ff29d29c94c2" containerName="discovery"
Oct 29 23:43:06.653424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.652610    5428 cpu_manager.go:414] "containerMap: removing container" podUID="8d42d182-5615-4c04-a4af-5d7aee1c4dba" containerName="payload-pre-processing"
Oct 29 23:43:06.653424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.652619    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d42d182-5615-4c04-a4af-5d7aee1c4dba" containerName="payload-pre-processing"
Oct 29 23:43:06.653424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.652642    5428 cpu_manager.go:414] "containerMap: removing container" podUID="5186bf82-c9e1-49e3-9dd0-f362ddce36ce" containerName="payload-processing"
Oct 29 23:43:06.653424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.652650    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5186bf82-c9e1-49e3-9dd0-f362ddce36ce" containerName="payload-processing"
Oct 29 23:43:06.653424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.652879    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="43df95b6-fe67-4edd-aeea-ff29d29c94c2" containerName="discovery"
Oct 29 23:43:06.653424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.652903    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d42d182-5615-4c04-a4af-5d7aee1c4dba" containerName="payload-pre-processing"
Oct 29 23:43:06.653424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.652916    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5186bf82-c9e1-49e3-9dd0-f362ddce36ce" containerName="payload-processing"
Oct 29 23:43:06.659480 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.658878    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"
Oct 29 23:43:06.661063 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.661041    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"]
Oct 29 23:43:06.667457 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.667436    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.675714 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.675679    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-55c3fc-serving-cert\""
Oct 29 23:43:06.686717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.686691    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"]
Oct 29 23:43:06.690008 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.689984    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-adc7d7-openshift-default-dockercfg-6789g\""
Oct 29 23:43:06.695049 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.695017    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"]
Oct 29 23:43:06.740963 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.740930    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-istio-data\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.741144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.740977    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38ef730-877a-49cd-98c2-64cbb7894032-openshift-service-ca\") pod \"maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d\" (UID: \"f38ef730-877a-49cd-98c2-64cbb7894032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"
Oct 29 23:43:06.741144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.741056    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27603afe-5712-43b2-aaed-056d472de92a-istio-podinfo\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.741334 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.741202    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-workload-certs\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.741334 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.741255    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-istio-envoy\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.741334 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.741293    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-workload-socket\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.741549 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.741467    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27603afe-5712-43b2-aaed-056d472de92a-istiod-ca-cert\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.742764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.742194    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38ef730-877a-49cd-98c2-64cbb7894032-metrics-certs\") pod \"maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d\" (UID: \"f38ef730-877a-49cd-98c2-64cbb7894032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"
Oct 29 23:43:06.742764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.742270    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38ef730-877a-49cd-98c2-64cbb7894032-maas-api-tls\") pod \"maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d\" (UID: \"f38ef730-877a-49cd-98c2-64cbb7894032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"
Oct 29 23:43:06.742764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.742418    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27603afe-5712-43b2-aaed-056d472de92a-istio-token\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.742764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.742451    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47zlq\" (UniqueName: \"kubernetes.io/projected/27603afe-5712-43b2-aaed-056d472de92a-kube-api-access-47zlq\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.742764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.742593    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk6t9\" (UniqueName: \"kubernetes.io/projected/f38ef730-877a-49cd-98c2-64cbb7894032-kube-api-access-fk6t9\") pod \"maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d\" (UID: \"f38ef730-877a-49cd-98c2-64cbb7894032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"
Oct 29 23:43:06.742764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.742620    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-credential-socket\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.743174 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.742795    5428 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-cacerts\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:06.743174 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.742830    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh66c\" (UniqueName: \"kubernetes.io/projected/43df95b6-fe67-4edd-aeea-ff29d29c94c2-kube-api-access-zh66c\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:06.743174 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.742844    5428 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/43df95b6-fe67-4edd-aeea-ff29d29c94c2-istio-csr-dns-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:06.843901 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.843869    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-workload-socket\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.844060 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.843908    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27603afe-5712-43b2-aaed-056d472de92a-istiod-ca-cert\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.844060 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.843972    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38ef730-877a-49cd-98c2-64cbb7894032-metrics-certs\") pod \"maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d\" (UID: \"f38ef730-877a-49cd-98c2-64cbb7894032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"
Oct 29 23:43:06.844060 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.843996    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38ef730-877a-49cd-98c2-64cbb7894032-maas-api-tls\") pod \"maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d\" (UID: \"f38ef730-877a-49cd-98c2-64cbb7894032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"
Oct 29 23:43:06.844238 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.844084    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27603afe-5712-43b2-aaed-056d472de92a-istio-token\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.844238 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.844113    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47zlq\" (UniqueName: \"kubernetes.io/projected/27603afe-5712-43b2-aaed-056d472de92a-kube-api-access-47zlq\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.844413 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.844278    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-workload-socket\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.844599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.844537    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk6t9\" (UniqueName: \"kubernetes.io/projected/f38ef730-877a-49cd-98c2-64cbb7894032-kube-api-access-fk6t9\") pod \"maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d\" (UID: \"f38ef730-877a-49cd-98c2-64cbb7894032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"
Oct 29 23:43:06.844731 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.844601    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-credential-socket\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.844731 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.844636    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27603afe-5712-43b2-aaed-056d472de92a-istiod-ca-cert\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.844731 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.844699    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-istio-data\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.844945 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.844731    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38ef730-877a-49cd-98c2-64cbb7894032-openshift-service-ca\") pod \"maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d\" (UID: \"f38ef730-877a-49cd-98c2-64cbb7894032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"
Oct 29 23:43:06.844945 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.844758    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27603afe-5712-43b2-aaed-056d472de92a-istio-podinfo\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.844945 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.844919    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-workload-certs\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.845158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.844985    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-istio-envoy\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.845158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.845021    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-istio-data\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.845158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.844917    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-credential-socket\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.845642 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.845583    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38ef730-877a-49cd-98c2-64cbb7894032-openshift-service-ca\") pod \"maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d\" (UID: \"f38ef730-877a-49cd-98c2-64cbb7894032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"
Oct 29 23:43:06.845971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.845951    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-workload-certs\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.847125 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.847104    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38ef730-877a-49cd-98c2-64cbb7894032-metrics-certs\") pod \"maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d\" (UID: \"f38ef730-877a-49cd-98c2-64cbb7894032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"
Oct 29 23:43:06.847210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.847155    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38ef730-877a-49cd-98c2-64cbb7894032-maas-api-tls\") pod \"maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d\" (UID: \"f38ef730-877a-49cd-98c2-64cbb7894032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"
Oct 29 23:43:06.847416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.847399    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-istio-envoy\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.847463 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.847446    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27603afe-5712-43b2-aaed-056d472de92a-istio-podinfo\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.867139 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.867111    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27603afe-5712-43b2-aaed-056d472de92a-istio-token\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.867291 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.867274    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47zlq\" (UniqueName: \"kubernetes.io/projected/27603afe-5712-43b2-aaed-056d472de92a-kube-api-access-47zlq\") pod \"e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") " pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:06.867917 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:06.867893    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk6t9\" (UniqueName: \"kubernetes.io/projected/f38ef730-877a-49cd-98c2-64cbb7894032-kube-api-access-fk6t9\") pod \"maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d\" (UID: \"f38ef730-877a-49cd-98c2-64cbb7894032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"
Oct 29 23:43:07.016246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.016211    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"
Oct 29 23:43:07.023876 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.023848    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:07.123606 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.123311    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:07.336359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.336286    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:07.358018 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.357988    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"]
Oct 29 23:43:07.358208 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:43:07.358185    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38ef730_877a_49cd_98c2_64cbb7894032.slice/crio-fd4ab3f364f2e6d6ad1a4e1a1d4c80d9d9812a5e183017315ad333cd06908225 WatchSource:0}: Error finding container fd4ab3f364f2e6d6ad1a4e1a1d4c80d9d9812a5e183017315ad333cd06908225: Status 404 returned error can't find the container with id fd4ab3f364f2e6d6ad1a4e1a1d4c80d9d9812a5e183017315ad333cd06908225
Oct 29 23:43:07.360633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.360614    5428 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:43:07.362051 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:43:07.362029    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27603afe_5712_43b2_aaed_056d472de92a.slice/crio-29077d5a405fdd27105c16bd47b038607034045568dcb95df9301c12f92b027b WatchSource:0}: Error finding container 29077d5a405fdd27105c16bd47b038607034045568dcb95df9301c12f92b027b: Status 404 returned error can't find the container with id 29077d5a405fdd27105c16bd47b038607034045568dcb95df9301c12f92b027b
Oct 29 23:43:07.364091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.364066    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"]
Oct 29 23:43:07.365148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.365117    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:43:07.365302 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.365194    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:43:07.365302 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.365245    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:43:07.467721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.467693    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:07.549227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.547532    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46" event={"ID":"43df95b6-fe67-4edd-aeea-ff29d29c94c2","Type":"ContainerDied","Data":"3a0b1734d413f599cdae7d6cf1e155fd0e567608de99152f0297540f80f1c7cd"}
Oct 29 23:43:07.549227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.547597    5428 scope.go:122] "RemoveContainer" containerID="8d086784ef382515f0d9a163d5af9cf9528550b9e7f2f1e84b1b051af6c44f0d"
Oct 29 23:43:07.549227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.547744    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"
Oct 29 23:43:07.553959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.551698    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d" event={"ID":"f38ef730-877a-49cd-98c2-64cbb7894032","Type":"ContainerStarted","Data":"fd4ab3f364f2e6d6ad1a4e1a1d4c80d9d9812a5e183017315ad333cd06908225"}
Oct 29 23:43:07.557899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.554988    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp" event={"ID":"27603afe-5712-43b2-aaed-056d472de92a","Type":"ContainerStarted","Data":"d4637797b8d50243bb72281afd00d1ca22f09fcb4e67ceda6cc7252bbd1c49f7"}
Oct 29 23:43:07.557899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.555036    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp" event={"ID":"27603afe-5712-43b2-aaed-056d472de92a","Type":"ContainerStarted","Data":"29077d5a405fdd27105c16bd47b038607034045568dcb95df9301c12f92b027b"}
Oct 29 23:43:07.641824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.634854    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"]
Oct 29 23:43:07.650819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.644012    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cxw46"]
Oct 29 23:43:07.714948 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.714689    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:07.745867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:07.745406    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp" podStartSLOduration=65.745385701 podStartE2EDuration="1m5.745385701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:43:07.69802095 +0000 UTC m=+3591.040969696" watchObservedRunningTime="2026-10-29 23:43:07.745385701 +0000 UTC m=+3591.088334447"
Oct 29 23:43:08.025890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:08.024796    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:08.029989 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:08.029959    5428 patch_prober.go:29] interesting pod/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 29 23:43:08.030347 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:08.030312    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp" podUID="27603afe-5712-43b2-aaed-056d472de92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 29 23:43:08.569282 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:08.569241    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d" event={"ID":"f38ef730-877a-49cd-98c2-64cbb7894032","Type":"ContainerStarted","Data":"f5393fcbecf5d6b18b53ec49053e19d9313dbab81a6f69e343572f7c614eb30e"}
Oct 29 23:43:08.569770 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:08.569338    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"
Oct 29 23:43:08.617898 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:08.617840    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d" podStartSLOduration=94.237718083 podStartE2EDuration="1m34.617821099s" totalImagesPullingTime="380.103016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:43:08.615915069 +0000 UTC m=+3591.958863814" watchObservedRunningTime="2026-10-29 23:43:08.617821099 +0000 UTC m=+3591.960769843"
Oct 29 23:43:08.781723 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:08.781691    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43df95b6-fe67-4edd-aeea-ff29d29c94c2" path="/var/lib/kubelet/pods/43df95b6-fe67-4edd-aeea-ff29d29c94c2/volumes"
Oct 29 23:43:09.030961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:09.030929    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:09.534116 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:09.534082    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:09.580199 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:09.580162    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:09.580884 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:09.580862    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:43:10.193941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:10.193908    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:10.314398 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:10.314368    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:11.539209 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:11.539176    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:12.062993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.062961    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:12.172619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.172588    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs"]
Oct 29 23:43:12.189901 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.189863    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs"
Oct 29 23:43:12.197624 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.197591    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lcnd\" (UniqueName: \"kubernetes.io/projected/85e2d99a-031b-441f-b881-df20fba27bba-kube-api-access-8lcnd\") pod \"payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs\" (UID: \"85e2d99a-031b-441f-b881-df20fba27bba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs"
Oct 29 23:43:12.198378 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.198349    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85e2d99a-031b-441f-b881-df20fba27bba-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs\" (UID: \"85e2d99a-031b-441f-b881-df20fba27bba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs"
Oct 29 23:43:12.205934 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.205910    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs"]
Oct 29 23:43:12.211006 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.210979    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-adc7d7\""
Oct 29 23:43:12.211105 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.210984    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:12.222837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.222813    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-adc7d7-dockercfg-t9h7f\""
Oct 29 23:43:12.228721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.228698    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh"]
Oct 29 23:43:12.234010 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.233993    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh"
Oct 29 23:43:12.252734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.252642    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh"]
Oct 29 23:43:12.310824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.300022    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lcnd\" (UniqueName: \"kubernetes.io/projected/85e2d99a-031b-441f-b881-df20fba27bba-kube-api-access-8lcnd\") pod \"payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs\" (UID: \"85e2d99a-031b-441f-b881-df20fba27bba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs"
Oct 29 23:43:12.310824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.300197    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh6cz\" (UniqueName: \"kubernetes.io/projected/e5d61a10-3f0c-42b8-a289-3c0830622387-kube-api-access-gh6cz\") pod \"payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh\" (UID: \"e5d61a10-3f0c-42b8-a289-3c0830622387\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh"
Oct 29 23:43:12.310824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.300419    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85e2d99a-031b-441f-b881-df20fba27bba-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs\" (UID: \"85e2d99a-031b-441f-b881-df20fba27bba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs"
Oct 29 23:43:12.310824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.300496    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d61a10-3f0c-42b8-a289-3c0830622387-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh\" (UID: \"e5d61a10-3f0c-42b8-a289-3c0830622387\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh"
Oct 29 23:43:12.310824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.301758    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85e2d99a-031b-441f-b881-df20fba27bba-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs\" (UID: \"85e2d99a-031b-441f-b881-df20fba27bba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs"
Oct 29 23:43:12.351868 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.346014    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"]
Oct 29 23:43:12.351868 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.346292    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj" podUID="bf911873-fc76-4358-8c69-b6d644701bfd" containerName="istio-proxy" containerID="cri-o://45bef36ee2dcfda2fd5a7e0b248ca63b252174ea434cb29274adc647137abe81" gracePeriod=30
Oct 29 23:43:12.365845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.354642    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lcnd\" (UniqueName: \"kubernetes.io/projected/85e2d99a-031b-441f-b881-df20fba27bba-kube-api-access-8lcnd\") pod \"payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs\" (UID: \"85e2d99a-031b-441f-b881-df20fba27bba\") " pod="openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs"
Oct 29 23:43:12.365845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.359028    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"]
Oct 29 23:43:12.365845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.359320    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j" podUID="4c6c8bc1-5531-4ae7-9660-67ee6263b34e" containerName="istio-proxy" containerID="cri-o://aa37b38e4737888ccc6e5c62449060fc8e09bf20d5de7c2f64187cdd147dccac" gracePeriod=30
Oct 29 23:43:12.431817 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.424341    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh6cz\" (UniqueName: \"kubernetes.io/projected/e5d61a10-3f0c-42b8-a289-3c0830622387-kube-api-access-gh6cz\") pod \"payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh\" (UID: \"e5d61a10-3f0c-42b8-a289-3c0830622387\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh"
Oct 29 23:43:12.431817 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.424612    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d61a10-3f0c-42b8-a289-3c0830622387-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh\" (UID: \"e5d61a10-3f0c-42b8-a289-3c0830622387\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh"
Oct 29 23:43:12.431817 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.425735    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d61a10-3f0c-42b8-a289-3c0830622387-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh\" (UID: \"e5d61a10-3f0c-42b8-a289-3c0830622387\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh"
Oct 29 23:43:12.450431 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.450368    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh6cz\" (UniqueName: \"kubernetes.io/projected/e5d61a10-3f0c-42b8-a289-3c0830622387-kube-api-access-gh6cz\") pod \"payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh\" (UID: \"e5d61a10-3f0c-42b8-a289-3c0830622387\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh"
Oct 29 23:43:12.521652 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.514754    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs"
Oct 29 23:43:12.577375 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.577340    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh"
Oct 29 23:43:12.882825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.882779    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs"]
Oct 29 23:43:12.884643 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:43:12.884605    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e2d99a_031b_441f_b881_df20fba27bba.slice/crio-b5979681ecf2ced5be19188d58c75f54215072f3cc73b10c77b781192b68b3cc WatchSource:0}: Error finding container b5979681ecf2ced5be19188d58c75f54215072f3cc73b10c77b781192b68b3cc: Status 404 returned error can't find the container with id b5979681ecf2ced5be19188d58c75f54215072f3cc73b10c77b781192b68b3cc
Oct 29 23:43:12.911182 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:12.911156    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh"]
Oct 29 23:43:12.922597 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:43:12.922560    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d61a10_3f0c_42b8_a289_3c0830622387.slice/crio-ea90e1ef50065d463a35c5b29f669bae31c33ce59b7943ca4cd843c3c3cdeca9 WatchSource:0}: Error finding container ea90e1ef50065d463a35c5b29f669bae31c33ce59b7943ca4cd843c3c3cdeca9: Status 404 returned error can't find the container with id ea90e1ef50065d463a35c5b29f669bae31c33ce59b7943ca4cd843c3c3cdeca9
Oct 29 23:43:13.579126 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:13.579086    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:13.647772 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:13.647731    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh" event={"ID":"e5d61a10-3f0c-42b8-a289-3c0830622387","Type":"ContainerStarted","Data":"ec1eb6dabcd3d09d128fde48fea76ac9a9430911954c5ecbc3a7f921aade5d4f"}
Oct 29 23:43:13.647979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:13.647899    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh"
Oct 29 23:43:13.647979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:13.647933    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh" event={"ID":"e5d61a10-3f0c-42b8-a289-3c0830622387","Type":"ContainerStarted","Data":"ea90e1ef50065d463a35c5b29f669bae31c33ce59b7943ca4cd843c3c3cdeca9"}
Oct 29 23:43:13.650012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:13.649983    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs" event={"ID":"85e2d99a-031b-441f-b881-df20fba27bba","Type":"ContainerStarted","Data":"b2e41f5010f5de32a01a592a9b4140ad68797292d418202e8361d6d457055837"}
Oct 29 23:43:13.650176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:13.650032    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs" event={"ID":"85e2d99a-031b-441f-b881-df20fba27bba","Type":"ContainerStarted","Data":"b5979681ecf2ced5be19188d58c75f54215072f3cc73b10c77b781192b68b3cc"}
Oct 29 23:43:13.650641 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:13.650622    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs"
Oct 29 23:43:13.692291 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:13.692233    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh" podStartSLOduration=1.6922149549999999 podStartE2EDuration="1.692214955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:43:13.687971335 +0000 UTC m=+3597.030920080" watchObservedRunningTime="2026-10-29 23:43:13.692214955 +0000 UTC m=+3597.035163702"
Oct 29 23:43:13.731716 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:13.731661    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs" podStartSLOduration=1.7316491859999998 podStartE2EDuration="1.731649186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:43:13.72827967 +0000 UTC m=+3597.071228419" watchObservedRunningTime="2026-10-29 23:43:13.731649186 +0000 UTC m=+3597.074597930"
Oct 29 23:43:14.587911 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:14.587880    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"
Oct 29 23:43:15.377106 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:15.377070    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:17.059631 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.059600    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:17.369998 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:43:17.369967    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:43:17.697304 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.697266    5428 generic.go:356] "Generic (PLEG): container finished" podID="4c6c8bc1-5531-4ae7-9660-67ee6263b34e" containerID="aa37b38e4737888ccc6e5c62449060fc8e09bf20d5de7c2f64187cdd147dccac" exitCode=0
Oct 29 23:43:17.697487 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.697334    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j" event={"ID":"4c6c8bc1-5531-4ae7-9660-67ee6263b34e","Type":"ContainerDied","Data":"aa37b38e4737888ccc6e5c62449060fc8e09bf20d5de7c2f64187cdd147dccac"}
Oct 29 23:43:17.699324 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.699297    5428 generic.go:356] "Generic (PLEG): container finished" podID="bf911873-fc76-4358-8c69-b6d644701bfd" containerID="45bef36ee2dcfda2fd5a7e0b248ca63b252174ea434cb29274adc647137abe81" exitCode=0
Oct 29 23:43:17.699448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.699369    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj" event={"ID":"bf911873-fc76-4358-8c69-b6d644701bfd","Type":"ContainerDied","Data":"45bef36ee2dcfda2fd5a7e0b248ca63b252174ea434cb29274adc647137abe81"}
Oct 29 23:43:17.751232 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.751199    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:43:17.833931 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.833895    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-podinfo\") pod \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") "
Oct 29 23:43:17.834124 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.833956    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-credential-socket\") pod \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") "
Oct 29 23:43:17.834124 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.834005    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-envoy\") pod \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") "
Oct 29 23:43:17.834124 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.834052    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-token\" (UniqueName: \"kubernetes.io/projected/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-token\") pod \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") "
Oct 29 23:43:17.834318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.834229    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-kube-api-access-7fcxw\" (UniqueName: \"kubernetes.io/projected/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-kube-api-access-7fcxw\") pod \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") "
Oct 29 23:43:17.834318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.834275    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-data\") pod \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") "
Oct 29 23:43:17.834318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.834310    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istiod-ca-cert\") pod \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") "
Oct 29 23:43:17.834489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.834354    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-workload-socket\") pod \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") "
Oct 29 23:43:17.834489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.834480    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-workload-certs\") pod \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\" (UID: \"4c6c8bc1-5531-4ae7-9660-67ee6263b34e\") "
Oct 29 23:43:17.836661 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.834962    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-workload-socket" pod "4c6c8bc1-5531-4ae7-9660-67ee6263b34e" (UID: "4c6c8bc1-5531-4ae7-9660-67ee6263b34e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:43:17.836661 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.835210    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-workload-certs" pod "4c6c8bc1-5531-4ae7-9660-67ee6263b34e" (UID: "4c6c8bc1-5531-4ae7-9660-67ee6263b34e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:43:17.836661 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.836063    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:17.836661 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.836087    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:17.836661 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.836332    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-credential-socket" pod "4c6c8bc1-5531-4ae7-9660-67ee6263b34e" (UID: "4c6c8bc1-5531-4ae7-9660-67ee6263b34e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:43:17.837906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.837868    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istiod-ca-cert" pod "4c6c8bc1-5531-4ae7-9660-67ee6263b34e" (UID: "4c6c8bc1-5531-4ae7-9660-67ee6263b34e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:43:17.838303 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.838281    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-data" pod "4c6c8bc1-5531-4ae7-9660-67ee6263b34e" (UID: "4c6c8bc1-5531-4ae7-9660-67ee6263b34e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:43:17.840879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.840852    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-kube-api-access-7fcxw" pod "4c6c8bc1-5531-4ae7-9660-67ee6263b34e" (UID: "4c6c8bc1-5531-4ae7-9660-67ee6263b34e"). InnerVolumeSpecName "kube-api-access-7fcxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:43:17.841561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.841537    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-podinfo" pod "4c6c8bc1-5531-4ae7-9660-67ee6263b34e" (UID: "4c6c8bc1-5531-4ae7-9660-67ee6263b34e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:43:17.842017 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.841992    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-envoy" pod "4c6c8bc1-5531-4ae7-9660-67ee6263b34e" (UID: "4c6c8bc1-5531-4ae7-9660-67ee6263b34e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:43:17.842474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.842441    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-token" pod "4c6c8bc1-5531-4ae7-9660-67ee6263b34e" (UID: "4c6c8bc1-5531-4ae7-9660-67ee6263b34e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:43:17.843007 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.842986    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:43:17.937172 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.937069    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf911873-fc76-4358-8c69-b6d644701bfd-kube-api-access-fkqwf\" (UniqueName: \"kubernetes.io/projected/bf911873-fc76-4358-8c69-b6d644701bfd-kube-api-access-fkqwf\") pod \"bf911873-fc76-4358-8c69-b6d644701bfd\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") "
Oct 29 23:43:17.937361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.937179    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-credential-socket\") pod \"bf911873-fc76-4358-8c69-b6d644701bfd\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") "
Oct 29 23:43:17.937361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.937325    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf911873-fc76-4358-8c69-b6d644701bfd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf911873-fc76-4358-8c69-b6d644701bfd-istio-podinfo\") pod \"bf911873-fc76-4358-8c69-b6d644701bfd\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") "
Oct 29 23:43:17.937670 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.937641    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-workload-socket\") pod \"bf911873-fc76-4358-8c69-b6d644701bfd\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") "
Oct 29 23:43:17.937784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.937693    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-workload-certs\") pod \"bf911873-fc76-4358-8c69-b6d644701bfd\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") "
Oct 29 23:43:17.937869 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.937833    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf911873-fc76-4358-8c69-b6d644701bfd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf911873-fc76-4358-8c69-b6d644701bfd-istiod-ca-cert\") pod \"bf911873-fc76-4358-8c69-b6d644701bfd\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") "
Oct 29 23:43:17.937936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.937892    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-istio-data\") pod \"bf911873-fc76-4358-8c69-b6d644701bfd\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") "
Oct 29 23:43:17.937993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.937966    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf911873-fc76-4358-8c69-b6d644701bfd-istio-token\" (UniqueName: \"kubernetes.io/projected/bf911873-fc76-4358-8c69-b6d644701bfd-istio-token\") pod \"bf911873-fc76-4358-8c69-b6d644701bfd\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") "
Oct 29 23:43:17.938097 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.938079    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-istio-envoy\") pod \"bf911873-fc76-4358-8c69-b6d644701bfd\" (UID: \"bf911873-fc76-4358-8c69-b6d644701bfd\") "
Oct 29 23:43:17.938281 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.938251    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-workload-socket" pod "bf911873-fc76-4358-8c69-b6d644701bfd" (UID: "bf911873-fc76-4358-8c69-b6d644701bfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:43:17.939078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.938669    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-credential-socket" pod "bf911873-fc76-4358-8c69-b6d644701bfd" (UID: "bf911873-fc76-4358-8c69-b6d644701bfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:43:17.939078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.938777    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-workload-certs" pod "bf911873-fc76-4358-8c69-b6d644701bfd" (UID: "bf911873-fc76-4358-8c69-b6d644701bfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:43:17.939078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.938833    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-istio-data" pod "bf911873-fc76-4358-8c69-b6d644701bfd" (UID: "bf911873-fc76-4358-8c69-b6d644701bfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:43:17.939078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.938957    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf911873-fc76-4358-8c69-b6d644701bfd-istiod-ca-cert" pod "bf911873-fc76-4358-8c69-b6d644701bfd" (UID: "bf911873-fc76-4358-8c69-b6d644701bfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:43:17.939078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.938986    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:17.939078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.939007    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:17.939078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.939021    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:17.939078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.939036    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:17.939078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.939051    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:17.939078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.939065    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:17.939078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.939079    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:17.939762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.939092    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fcxw\" (UniqueName: \"kubernetes.io/projected/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-kube-api-access-7fcxw\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:17.939762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.939106    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:17.939762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.939119    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6c8bc1-5531-4ae7-9660-67ee6263b34e-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:17.939762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.939131    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:17.940002 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.939976    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf911873-fc76-4358-8c69-b6d644701bfd-kube-api-access-fkqwf" pod "bf911873-fc76-4358-8c69-b6d644701bfd" (UID: "bf911873-fc76-4358-8c69-b6d644701bfd"). InnerVolumeSpecName "kube-api-access-fkqwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:43:17.941096 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.941072    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-istio-envoy" pod "bf911873-fc76-4358-8c69-b6d644701bfd" (UID: "bf911873-fc76-4358-8c69-b6d644701bfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:43:17.941333 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.941313    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf911873-fc76-4358-8c69-b6d644701bfd-istio-podinfo" pod "bf911873-fc76-4358-8c69-b6d644701bfd" (UID: "bf911873-fc76-4358-8c69-b6d644701bfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:43:17.941545 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:17.941522    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf911873-fc76-4358-8c69-b6d644701bfd-istio-token" pod "bf911873-fc76-4358-8c69-b6d644701bfd" (UID: "bf911873-fc76-4358-8c69-b6d644701bfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:43:18.040795 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:18.040765    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkqwf\" (UniqueName: \"kubernetes.io/projected/bf911873-fc76-4358-8c69-b6d644701bfd-kube-api-access-fkqwf\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:18.040795 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:18.040791    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf911873-fc76-4358-8c69-b6d644701bfd-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:18.041030 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:18.040835    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf911873-fc76-4358-8c69-b6d644701bfd-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:18.041030 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:18.040852    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf911873-fc76-4358-8c69-b6d644701bfd-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:18.041030 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:18.040864    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf911873-fc76-4358-8c69-b6d644701bfd-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:18.710065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:18.710027    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"
Oct 29 23:43:18.710065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:18.710050    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j" event={"ID":"4c6c8bc1-5531-4ae7-9660-67ee6263b34e","Type":"ContainerDied","Data":"a7fb930c520e8069614e517da54eced19728cf07b84456142e6dc918e1a6ecb9"}
Oct 29 23:43:18.710597 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:18.710117    5428 scope.go:122] "RemoveContainer" containerID="aa37b38e4737888ccc6e5c62449060fc8e09bf20d5de7c2f64187cdd147dccac"
Oct 29 23:43:18.712254 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:18.712226    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj" event={"ID":"bf911873-fc76-4358-8c69-b6d644701bfd","Type":"ContainerDied","Data":"44d9bf05926f029f4b73deea3fb59c1f7e5164671b6fbd22d55eb6b92721b900"}
Oct 29 23:43:18.712403 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:18.712300    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"
Oct 29 23:43:18.723370 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:18.723355    5428 scope.go:122] "RemoveContainer" containerID="45bef36ee2dcfda2fd5a7e0b248ca63b252174ea434cb29274adc647137abe81"
Oct 29 23:43:18.780187 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:18.780161    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"]
Oct 29 23:43:18.782196 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:18.782162    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:18.791600 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:18.791577    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-vdxdj"]
Oct 29 23:43:18.836825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:18.835748    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"]
Oct 29 23:43:18.843213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:18.843168    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-5fg8j"]
Oct 29 23:43:18.981068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:18.981034    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt"]
Oct 29 23:43:19.087363 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:19.087275    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:19.207685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:19.207654    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:20.749417 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:20.749387    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:20.780066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:20.780035    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c6c8bc1-5531-4ae7-9660-67ee6263b34e" path="/var/lib/kubelet/pods/4c6c8bc1-5531-4ae7-9660-67ee6263b34e/volumes"
Oct 29 23:43:20.780682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:20.780662    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf911873-fc76-4358-8c69-b6d644701bfd" path="/var/lib/kubelet/pods/bf911873-fc76-4358-8c69-b6d644701bfd/volumes"
Oct 29 23:43:20.883542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:20.883511    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:22.894971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:22.894935    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:23.981789 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:23.981752    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt" podUID="7bb46ed8-8d72-48c9-baa5-543d8a1824fb" containerName="payload-pre-processing" containerID="cri-o://039052ba8a003a1945e28bade27f22594c9459d2335acb7f9b5de54d1d990620" gracePeriod=25
Oct 29 23:43:24.313164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:24.313127    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt"
Oct 29 23:43:24.347175 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:24.347148    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb46ed8-8d72-48c9-baa5-543d8a1824fb-kube-api-access-26sh2\" (UniqueName: \"kubernetes.io/projected/7bb46ed8-8d72-48c9-baa5-543d8a1824fb-kube-api-access-26sh2\") pod \"7bb46ed8-8d72-48c9-baa5-543d8a1824fb\" (UID: \"7bb46ed8-8d72-48c9-baa5-543d8a1824fb\") "
Oct 29 23:43:24.347410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:24.347387    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bb46ed8-8d72-48c9-baa5-543d8a1824fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb46ed8-8d72-48c9-baa5-543d8a1824fb-plugins-config-volume\") pod \"7bb46ed8-8d72-48c9-baa5-543d8a1824fb\" (UID: \"7bb46ed8-8d72-48c9-baa5-543d8a1824fb\") "
Oct 29 23:43:24.347866 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:24.347835    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb46ed8-8d72-48c9-baa5-543d8a1824fb-plugins-config-volume" pod "7bb46ed8-8d72-48c9-baa5-543d8a1824fb" (UID: "7bb46ed8-8d72-48c9-baa5-543d8a1824fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:43:24.348134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:24.348114    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb46ed8-8d72-48c9-baa5-543d8a1824fb-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:24.349759 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:24.349715    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb46ed8-8d72-48c9-baa5-543d8a1824fb-kube-api-access-26sh2" pod "7bb46ed8-8d72-48c9-baa5-543d8a1824fb" (UID: "7bb46ed8-8d72-48c9-baa5-543d8a1824fb"). InnerVolumeSpecName "kube-api-access-26sh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:43:24.449570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:24.449526    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26sh2\" (UniqueName: \"kubernetes.io/projected/7bb46ed8-8d72-48c9-baa5-543d8a1824fb-kube-api-access-26sh2\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:24.665345 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:24.665317    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh"
Oct 29 23:43:24.668658 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:24.668630    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:24.773376 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:24.773341    5428 generic.go:356] "Generic (PLEG): container finished" podID="7bb46ed8-8d72-48c9-baa5-543d8a1824fb" containerID="039052ba8a003a1945e28bade27f22594c9459d2335acb7f9b5de54d1d990620" exitCode=0
Oct 29 23:43:24.775160 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:24.773492    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt"
Oct 29 23:43:24.784220 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:24.784194    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt" event={"ID":"7bb46ed8-8d72-48c9-baa5-543d8a1824fb","Type":"ContainerDied","Data":"039052ba8a003a1945e28bade27f22594c9459d2335acb7f9b5de54d1d990620"}
Oct 29 23:43:24.784331 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:24.784225    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt" event={"ID":"7bb46ed8-8d72-48c9-baa5-543d8a1824fb","Type":"ContainerDied","Data":"0342ee334d1314f749bdb5e0fe1f967fec8072d78e3a8023c920d9a69d58211a"}
Oct 29 23:43:24.784331 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:24.784251    5428 scope.go:122] "RemoveContainer" containerID="039052ba8a003a1945e28bade27f22594c9459d2335acb7f9b5de54d1d990620"
Oct 29 23:43:24.796950 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:24.796926    5428 scope.go:122] "RemoveContainer" containerID="039052ba8a003a1945e28bade27f22594c9459d2335acb7f9b5de54d1d990620"
Oct 29 23:43:24.797209 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:43:24.797190    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"039052ba8a003a1945e28bade27f22594c9459d2335acb7f9b5de54d1d990620\": container with ID starting with 039052ba8a003a1945e28bade27f22594c9459d2335acb7f9b5de54d1d990620 not found: ID does not exist" containerID="039052ba8a003a1945e28bade27f22594c9459d2335acb7f9b5de54d1d990620"
Oct 29 23:43:24.797268 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:24.797216    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"039052ba8a003a1945e28bade27f22594c9459d2335acb7f9b5de54d1d990620"} err="failed to get container status \"039052ba8a003a1945e28bade27f22594c9459d2335acb7f9b5de54d1d990620\": rpc error: code = NotFound desc = could not find container \"039052ba8a003a1945e28bade27f22594c9459d2335acb7f9b5de54d1d990620\": container with ID starting with 039052ba8a003a1945e28bade27f22594c9459d2335acb7f9b5de54d1d990620 not found: ID does not exist"
Oct 29 23:43:24.829734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:24.829703    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt"]
Oct 29 23:43:24.834508 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:24.834484    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d5e482-795dd94dff-pglnt"]
Oct 29 23:43:25.675098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:25.675070    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs"
Oct 29 23:43:26.460659 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:26.460617    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:26.782778 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:26.782706    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bb46ed8-8d72-48c9-baa5-543d8a1824fb" path="/var/lib/kubelet/pods/7bb46ed8-8d72-48c9-baa5-543d8a1824fb/volumes"
Oct 29 23:43:28.053398 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:28.053321    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:29.772396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:29.772362    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:30.480898 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:30.480862    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69"]
Oct 29 23:43:30.642880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:30.642821    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:30.762357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:30.762261    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:31.670252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.670211    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"]
Oct 29 23:43:31.671395 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.671372    5428 cpu_manager.go:414] "containerMap: removing container" podUID="bf911873-fc76-4358-8c69-b6d644701bfd" containerName="istio-proxy"
Oct 29 23:43:31.671395 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.671394    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf911873-fc76-4358-8c69-b6d644701bfd" containerName="istio-proxy"
Oct 29 23:43:31.671598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.671418    5428 cpu_manager.go:414] "containerMap: removing container" podUID="4c6c8bc1-5531-4ae7-9660-67ee6263b34e" containerName="istio-proxy"
Oct 29 23:43:31.671598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.671427    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c6c8bc1-5531-4ae7-9660-67ee6263b34e" containerName="istio-proxy"
Oct 29 23:43:31.671598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.671465    5428 cpu_manager.go:414] "containerMap: removing container" podUID="7bb46ed8-8d72-48c9-baa5-543d8a1824fb" containerName="payload-pre-processing"
Oct 29 23:43:31.671598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.671474    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb46ed8-8d72-48c9-baa5-543d8a1824fb" containerName="payload-pre-processing"
Oct 29 23:43:31.671826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.671756    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf911873-fc76-4358-8c69-b6d644701bfd" containerName="istio-proxy"
Oct 29 23:43:31.671826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.671773    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c6c8bc1-5531-4ae7-9660-67ee6263b34e" containerName="istio-proxy"
Oct 29 23:43:31.671826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.671787    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb46ed8-8d72-48c9-baa5-543d8a1824fb" containerName="payload-pre-processing"
Oct 29 23:43:31.677607 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.677586    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"
Oct 29 23:43:31.698762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.698733    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-825472-serving-cert\""
Oct 29 23:43:31.701623 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.701595    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"]
Oct 29 23:43:31.769463 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.769427    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d6288f-a859-4dee-9fc8-389435a62863-maas-api-tls\") pod \"maas-api-e2e-inf-b-825472-5b95789775-q6stt\" (UID: \"b6d6288f-a859-4dee-9fc8-389435a62863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"
Oct 29 23:43:31.769635 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.769491    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d6288f-a859-4dee-9fc8-389435a62863-metrics-certs\") pod \"maas-api-e2e-inf-b-825472-5b95789775-q6stt\" (UID: \"b6d6288f-a859-4dee-9fc8-389435a62863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"
Oct 29 23:43:31.769688 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.769650    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d6288f-a859-4dee-9fc8-389435a62863-openshift-service-ca\") pod \"maas-api-e2e-inf-b-825472-5b95789775-q6stt\" (UID: \"b6d6288f-a859-4dee-9fc8-389435a62863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"
Oct 29 23:43:31.769863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.769833    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tzk5\" (UniqueName: \"kubernetes.io/projected/b6d6288f-a859-4dee-9fc8-389435a62863-kube-api-access-5tzk5\") pod \"maas-api-e2e-inf-b-825472-5b95789775-q6stt\" (UID: \"b6d6288f-a859-4dee-9fc8-389435a62863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"
Oct 29 23:43:31.786046 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.786011    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:31.871602 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.871568    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tzk5\" (UniqueName: \"kubernetes.io/projected/b6d6288f-a859-4dee-9fc8-389435a62863-kube-api-access-5tzk5\") pod \"maas-api-e2e-inf-b-825472-5b95789775-q6stt\" (UID: \"b6d6288f-a859-4dee-9fc8-389435a62863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"
Oct 29 23:43:31.872043 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.871708    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d6288f-a859-4dee-9fc8-389435a62863-maas-api-tls\") pod \"maas-api-e2e-inf-b-825472-5b95789775-q6stt\" (UID: \"b6d6288f-a859-4dee-9fc8-389435a62863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"
Oct 29 23:43:31.872043 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.871757    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d6288f-a859-4dee-9fc8-389435a62863-metrics-certs\") pod \"maas-api-e2e-inf-b-825472-5b95789775-q6stt\" (UID: \"b6d6288f-a859-4dee-9fc8-389435a62863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"
Oct 29 23:43:31.872457 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.872422    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d6288f-a859-4dee-9fc8-389435a62863-openshift-service-ca\") pod \"maas-api-e2e-inf-b-825472-5b95789775-q6stt\" (UID: \"b6d6288f-a859-4dee-9fc8-389435a62863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"
Oct 29 23:43:31.873487 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.873464    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d6288f-a859-4dee-9fc8-389435a62863-openshift-service-ca\") pod \"maas-api-e2e-inf-b-825472-5b95789775-q6stt\" (UID: \"b6d6288f-a859-4dee-9fc8-389435a62863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"
Oct 29 23:43:31.876813 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.876765    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d6288f-a859-4dee-9fc8-389435a62863-maas-api-tls\") pod \"maas-api-e2e-inf-b-825472-5b95789775-q6stt\" (UID: \"b6d6288f-a859-4dee-9fc8-389435a62863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"
Oct 29 23:43:31.878725 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.878673    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d6288f-a859-4dee-9fc8-389435a62863-metrics-certs\") pod \"maas-api-e2e-inf-b-825472-5b95789775-q6stt\" (UID: \"b6d6288f-a859-4dee-9fc8-389435a62863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"
Oct 29 23:43:31.895939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.895907    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tzk5\" (UniqueName: \"kubernetes.io/projected/b6d6288f-a859-4dee-9fc8-389435a62863-kube-api-access-5tzk5\") pod \"maas-api-e2e-inf-b-825472-5b95789775-q6stt\" (UID: \"b6d6288f-a859-4dee-9fc8-389435a62863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"
Oct 29 23:43:31.901712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:31.901688    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:32.002099 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:32.002025    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"
Oct 29 23:43:32.013604 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:32.013576    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:32.325825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:32.325785    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"]
Oct 29 23:43:32.326489 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:43:32.326459    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d6288f_a859_4dee_9fc8_389435a62863.slice/crio-dcf8c9a0c9f06fee439ade6b89806e8432d8201dbd292ed9d0dcbe7fc7398afb WatchSource:0}: Error finding container dcf8c9a0c9f06fee439ade6b89806e8432d8201dbd292ed9d0dcbe7fc7398afb: Status 404 returned error can't find the container with id dcf8c9a0c9f06fee439ade6b89806e8432d8201dbd292ed9d0dcbe7fc7398afb
Oct 29 23:43:32.703513 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:32.703477    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:32.871843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:32.868252    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt" event={"ID":"b6d6288f-a859-4dee-9fc8-389435a62863","Type":"ContainerStarted","Data":"81c6ee577368aadc57f738f40ac1706057157a93780b05ce2eb4bda53ba953aa"}
Oct 29 23:43:32.871843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:32.868320    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt" event={"ID":"b6d6288f-a859-4dee-9fc8-389435a62863","Type":"ContainerStarted","Data":"dcf8c9a0c9f06fee439ade6b89806e8432d8201dbd292ed9d0dcbe7fc7398afb"}
Oct 29 23:43:32.871843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:32.868354    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"
Oct 29 23:43:32.914414 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:32.914360    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt" podStartSLOduration=1.5714516120000002 podStartE2EDuration="1.914342205s" totalImagesPullingTime="342.890593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:43:32.909209247 +0000 UTC m=+3616.252157993" watchObservedRunningTime="2026-10-29 23:43:32.914342205 +0000 UTC m=+3616.257290987"
Oct 29 23:43:33.423722 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:33.423682    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:33.675068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:33.674972    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:35.468961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:35.468927    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:35.481870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:35.481838    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69" podUID="69d00263-4608-4899-bc32-f3a966213cb9" containerName="payload-processing" containerID="cri-o://ca0bc6d633e463c52516cca75034ed4128b5438bb5208b97075a045984f61d07" gracePeriod=25
Oct 29 23:43:35.783115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:35.783087    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69"
Oct 29 23:43:35.899575 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:35.899538    5428 generic.go:356] "Generic (PLEG): container finished" podID="69d00263-4608-4899-bc32-f3a966213cb9" containerID="ca0bc6d633e463c52516cca75034ed4128b5438bb5208b97075a045984f61d07" exitCode=0
Oct 29 23:43:35.899742 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:35.899590    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69"
Oct 29 23:43:35.899742 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:35.899624    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69" event={"ID":"69d00263-4608-4899-bc32-f3a966213cb9","Type":"ContainerDied","Data":"ca0bc6d633e463c52516cca75034ed4128b5438bb5208b97075a045984f61d07"}
Oct 29 23:43:35.899742 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:35.899680    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69" event={"ID":"69d00263-4608-4899-bc32-f3a966213cb9","Type":"ContainerDied","Data":"8d0603584a7b39866af31cc39e50b06d0653f3d9c6cb25a2654a024ef9653fbe"}
Oct 29 23:43:35.899742 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:35.899711    5428 scope.go:122] "RemoveContainer" containerID="ca0bc6d633e463c52516cca75034ed4128b5438bb5208b97075a045984f61d07"
Oct 29 23:43:35.911865 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:35.911846    5428 scope.go:122] "RemoveContainer" containerID="ca0bc6d633e463c52516cca75034ed4128b5438bb5208b97075a045984f61d07"
Oct 29 23:43:35.912136 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:43:35.912118    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0bc6d633e463c52516cca75034ed4128b5438bb5208b97075a045984f61d07\": container with ID starting with ca0bc6d633e463c52516cca75034ed4128b5438bb5208b97075a045984f61d07 not found: ID does not exist" containerID="ca0bc6d633e463c52516cca75034ed4128b5438bb5208b97075a045984f61d07"
Oct 29 23:43:35.912184 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:35.912143    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0bc6d633e463c52516cca75034ed4128b5438bb5208b97075a045984f61d07"} err="failed to get container status \"ca0bc6d633e463c52516cca75034ed4128b5438bb5208b97075a045984f61d07\": rpc error: code = NotFound desc = could not find container \"ca0bc6d633e463c52516cca75034ed4128b5438bb5208b97075a045984f61d07\": container with ID starting with ca0bc6d633e463c52516cca75034ed4128b5438bb5208b97075a045984f61d07 not found: ID does not exist"
Oct 29 23:43:35.957943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:35.957923    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69d00263-4608-4899-bc32-f3a966213cb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d00263-4608-4899-bc32-f3a966213cb9-plugins-config-volume\") pod \"69d00263-4608-4899-bc32-f3a966213cb9\" (UID: \"69d00263-4608-4899-bc32-f3a966213cb9\") "
Oct 29 23:43:35.958138 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:35.958126    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d00263-4608-4899-bc32-f3a966213cb9-kube-api-access-czkxz\" (UniqueName: \"kubernetes.io/projected/69d00263-4608-4899-bc32-f3a966213cb9-kube-api-access-czkxz\") pod \"69d00263-4608-4899-bc32-f3a966213cb9\" (UID: \"69d00263-4608-4899-bc32-f3a966213cb9\") "
Oct 29 23:43:35.958273 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:35.958254    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d00263-4608-4899-bc32-f3a966213cb9-plugins-config-volume" pod "69d00263-4608-4899-bc32-f3a966213cb9" (UID: "69d00263-4608-4899-bc32-f3a966213cb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:43:35.959036 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:35.959008    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d00263-4608-4899-bc32-f3a966213cb9-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:35.960311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:35.960289    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d00263-4608-4899-bc32-f3a966213cb9-kube-api-access-czkxz" pod "69d00263-4608-4899-bc32-f3a966213cb9" (UID: "69d00263-4608-4899-bc32-f3a966213cb9"). InnerVolumeSpecName "kube-api-access-czkxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:43:36.060293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:36.060223    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czkxz\" (UniqueName: \"kubernetes.io/projected/69d00263-4608-4899-bc32-f3a966213cb9-kube-api-access-czkxz\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:43:36.251210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:36.251165    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69"]
Oct 29 23:43:36.262870 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:36.262836    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a8ab3c-76577d4796-9ns69"]
Oct 29 23:43:36.787141 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:36.787106    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69d00263-4608-4899-bc32-f3a966213cb9" path="/var/lib/kubelet/pods/69d00263-4608-4899-bc32-f3a966213cb9/volumes"
Oct 29 23:43:37.025999 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:37.025958    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:38.837205 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:38.837167    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:38.889052 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:38.889020    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"
Oct 29 23:43:40.597410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:40.597365    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:42.398167 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:42.398124    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:44.055474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:44.055443    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:46.139747 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:46.139716    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:46.814109 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:46.814071    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:47.376722 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:43:47.376685    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:43:48.193550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:48.193518    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:50.369021 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:50.368982    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:50.503333 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:50.503293    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:51.269983 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:51.269931    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:52.382237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:52.382178    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:54.390689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:54.390653    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:55.983606 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:55.983565    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:57.591650 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:57.591591    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:57.710314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:57.710266    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:57.826694 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:57.826662    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:43:59.502645 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:43:59.502609    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:01.300523 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:01.300474    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:02.916135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:02.916104    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:04.236750 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:04.236715    5428 generic.go:356] "Generic (PLEG): container finished" podID="97ec11c4-83fa-4ae5-94fa-35a14eed785f" containerID="59022cec8fa426e35fe84a9f22ea937326357091f5dcdde4b46ad01a08bc558b" exitCode=0
Oct 29 23:44:04.237238 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:04.236786    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr" event={"ID":"97ec11c4-83fa-4ae5-94fa-35a14eed785f","Type":"ContainerDied","Data":"59022cec8fa426e35fe84a9f22ea937326357091f5dcdde4b46ad01a08bc558b"}
Oct 29 23:44:04.237318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:04.237259    5428 scope.go:122] "RemoveContainer" containerID="59022cec8fa426e35fe84a9f22ea937326357091f5dcdde4b46ad01a08bc558b"
Oct 29 23:44:04.239472 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:04.239441    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:44:04.239605 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:04.239526    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:44:04.239605 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:04.239578    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:44:04.694621 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:04.694588    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:05.247662 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:05.247626    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr" event={"ID":"97ec11c4-83fa-4ae5-94fa-35a14eed785f","Type":"ContainerStarted","Data":"6d28c6b1a570b2d39a78079bc2eb077dbd8fa383f6c04ef7323de76b04d9e893"}
Oct 29 23:44:05.878825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:05.868201    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:44:05.882822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:05.879934    5428 patch_prober.go:29] interesting pod/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 29 23:44:05.882822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:05.879999    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr" podUID="97ec11c4-83fa-4ae5-94fa-35a14eed785f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 29 23:44:06.283892 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:06.280302    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:06.870179 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:06.870144    5428 patch_prober.go:29] interesting pod/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 29 23:44:06.870494 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:06.870462    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr" podUID="97ec11c4-83fa-4ae5-94fa-35a14eed785f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 29 23:44:07.880827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:07.877222    5428 patch_prober.go:29] interesting pod/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 29 23:44:07.880827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:07.877289    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr" podUID="97ec11c4-83fa-4ae5-94fa-35a14eed785f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 29 23:44:08.085182 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:08.081258    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:08.884361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:08.884319    5428 patch_prober.go:29] interesting pod/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 29 23:44:08.885003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:08.884393    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr" podUID="97ec11c4-83fa-4ae5-94fa-35a14eed785f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 29 23:44:09.583966 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:09.579284    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:09.869259 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:09.869224    5428 patch_prober.go:29] interesting pod/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 29 23:44:09.869541 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:09.869511    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr" podUID="97ec11c4-83fa-4ae5-94fa-35a14eed785f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 29 23:44:10.052877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:10.052846    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:44:10.375707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:10.375672    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:10.535225 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:10.535193    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:10.700251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:10.700156    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:10.940444 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:10.940401    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:44:10.941716 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:10.941685    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:44:11.097073 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:11.096984    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:11.243203 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:11.243158    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:12.545040 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:12.545006    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:13.674109 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.674069    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"]
Oct 29 23:44:13.675337 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.675313    5428 cpu_manager.go:414] "containerMap: removing container" podUID="69d00263-4608-4899-bc32-f3a966213cb9" containerName="payload-processing"
Oct 29 23:44:13.675337 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.675337    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d00263-4608-4899-bc32-f3a966213cb9" containerName="payload-processing"
Oct 29 23:44:13.675556 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.675541    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d00263-4608-4899-bc32-f3a966213cb9" containerName="payload-processing"
Oct 29 23:44:13.679893 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.679868    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:13.708388 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.708357    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-750972\"/\"openshift-service-ca.crt\""
Oct 29 23:44:13.708518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.708381    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-750972\"/\"kube-root-ca.crt\""
Oct 29 23:44:13.708518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.708387    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-750972\"/\"test-model-750972-kserve-self-signed-certs\""
Oct 29 23:44:13.708518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.708453    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-750972\"/\"default-dockercfg-psgz6\""
Oct 29 23:44:13.713576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.713551    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"]
Oct 29 23:44:13.732119 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.732094    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:13.761211 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.761187    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-model-cache\") pod \"test-model-750972-kserve-56f665676c-6s56t\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") " pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:13.761328 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.761225    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v472\" (UniqueName: \"kubernetes.io/projected/ad7987fc-c841-4693-94e3-36c855add12c-kube-api-access-9v472\") pod \"test-model-750972-kserve-56f665676c-6s56t\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") " pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:13.761328 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.761253    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-dshm\") pod \"test-model-750972-kserve-56f665676c-6s56t\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") " pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:13.761411 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.761363    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-tmp-dir\") pod \"test-model-750972-kserve-56f665676c-6s56t\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") " pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:13.761411 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.761398    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-home\") pod \"test-model-750972-kserve-56f665676c-6s56t\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") " pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:13.761797 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.761659    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad7987fc-c841-4693-94e3-36c855add12c-tls-certs\") pod \"test-model-750972-kserve-56f665676c-6s56t\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") " pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:13.862696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.862662    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-tmp-dir\") pod \"test-model-750972-kserve-56f665676c-6s56t\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") " pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:13.862696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.862710    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-home\") pod \"test-model-750972-kserve-56f665676c-6s56t\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") " pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:13.863033 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.863012    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad7987fc-c841-4693-94e3-36c855add12c-tls-certs\") pod \"test-model-750972-kserve-56f665676c-6s56t\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") " pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:13.863147 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.863130    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-model-cache\") pod \"test-model-750972-kserve-56f665676c-6s56t\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") " pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:13.863217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.863162    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-tmp-dir\") pod \"test-model-750972-kserve-56f665676c-6s56t\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") " pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:13.863217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.863183    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v472\" (UniqueName: \"kubernetes.io/projected/ad7987fc-c841-4693-94e3-36c855add12c-kube-api-access-9v472\") pod \"test-model-750972-kserve-56f665676c-6s56t\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") " pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:13.863329 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.863219    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-dshm\") pod \"test-model-750972-kserve-56f665676c-6s56t\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") " pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:13.863329 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.863228    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-home\") pod \"test-model-750972-kserve-56f665676c-6s56t\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") " pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:13.863778 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.863750    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-model-cache\") pod \"test-model-750972-kserve-56f665676c-6s56t\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") " pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:13.865518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.865490    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-dshm\") pod \"test-model-750972-kserve-56f665676c-6s56t\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") " pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:13.865665 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.865641    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad7987fc-c841-4693-94e3-36c855add12c-tls-certs\") pod \"test-model-750972-kserve-56f665676c-6s56t\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") " pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:13.886766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:13.886733    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v472\" (UniqueName: \"kubernetes.io/projected/ad7987fc-c841-4693-94e3-36c855add12c-kube-api-access-9v472\") pod \"test-model-750972-kserve-56f665676c-6s56t\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") " pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:14.013642 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:14.005168    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:14.039939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:14.039901    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:14.388901 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:14.388869    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"]
Oct 29 23:44:14.389965 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:44:14.389936    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad7987fc_c841_4693_94e3_36c855add12c.slice/crio-54a47aa5343230452c9ec6484924ef59b803cbff4ab7bc5fab9930d22eec8216 WatchSource:0}: Error finding container 54a47aa5343230452c9ec6484924ef59b803cbff4ab7bc5fab9930d22eec8216: Status 404 returned error can't find the container with id 54a47aa5343230452c9ec6484924ef59b803cbff4ab7bc5fab9930d22eec8216
Oct 29 23:44:15.007943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:15.007910    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:15.142603 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:15.142574    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:15.258130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:15.258038    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:15.372562 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:15.372526    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t" event={"ID":"ad7987fc-c841-4693-94e3-36c855add12c","Type":"ContainerStarted","Data":"2e707cbf2defea8b3199d2015195f8ddf339927c2b868a73f17624e4e061a1ff"}
Oct 29 23:44:15.372772 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:15.372577    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t" event={"ID":"ad7987fc-c841-4693-94e3-36c855add12c","Type":"ContainerStarted","Data":"54a47aa5343230452c9ec6484924ef59b803cbff4ab7bc5fab9930d22eec8216"}
Oct 29 23:44:15.415192 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:15.415140    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t" podStartSLOduration=2.415127212 podStartE2EDuration="2.415127212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:44:15.413013538 +0000 UTC m=+3658.755962284" watchObservedRunningTime="2026-10-29 23:44:15.415127212 +0000 UTC m=+3658.758075957"
Oct 29 23:44:15.657185 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:15.657155    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:16.005776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:16.005688    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:17.184713 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:17.184679    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:17.314904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:17.314872    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:17.369693 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:44:17.369659    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:44:17.445010 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:17.444907    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:19.302169 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:19.302130    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:19.899675 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:19.899632    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:21.057848 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:21.057786    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:22.377424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:22.377392    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:22.499726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:22.499695    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:22.695424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:22.695286    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"]
Oct 29 23:44:22.695634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:22.695529    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf" podUID="abd54fa0-a337-47a7-b507-7076286d9963" containerName="istio-proxy" containerID="cri-o://6e032b1b732ec57224d54ee1a3f883cc45cc77f9e8d8332f27124a4c198f792e" gracePeriod=30
Oct 29 23:44:22.718681 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:22.718653    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:23.503743 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:23.503702    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:24.006769 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:24.006736    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:24.020650 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:24.020620    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:24.470604 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:24.470573    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:44:25.522636 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:25.522600    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:27.272118 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:27.272087    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:27.558053 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:27.557979    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:27.991585 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:27.991561    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:44:28.103639 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.103547    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-workload-socket\") pod \"abd54fa0-a337-47a7-b507-7076286d9963\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") "
Oct 29 23:44:28.103896 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.103865    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-workload-socket" pod "abd54fa0-a337-47a7-b507-7076286d9963" (UID: "abd54fa0-a337-47a7-b507-7076286d9963"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:44:28.104078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.104054    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd54fa0-a337-47a7-b507-7076286d9963-istio-token\" (UniqueName: \"kubernetes.io/projected/abd54fa0-a337-47a7-b507-7076286d9963-istio-token\") pod \"abd54fa0-a337-47a7-b507-7076286d9963\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") "
Oct 29 23:44:28.104177 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.104143    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd54fa0-a337-47a7-b507-7076286d9963-kube-api-access-2pkzc\" (UniqueName: \"kubernetes.io/projected/abd54fa0-a337-47a7-b507-7076286d9963-kube-api-access-2pkzc\") pod \"abd54fa0-a337-47a7-b507-7076286d9963\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") "
Oct 29 23:44:28.104357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.104338    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-istio-data\") pod \"abd54fa0-a337-47a7-b507-7076286d9963\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") "
Oct 29 23:44:28.104575 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.104555    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-workload-certs\") pod \"abd54fa0-a337-47a7-b507-7076286d9963\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") "
Oct 29 23:44:28.104717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.104700    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-credential-socket\") pod \"abd54fa0-a337-47a7-b507-7076286d9963\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") "
Oct 29 23:44:28.104888 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.104864    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-istio-data" pod "abd54fa0-a337-47a7-b507-7076286d9963" (UID: "abd54fa0-a337-47a7-b507-7076286d9963"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:44:28.105030 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.105010    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abd54fa0-a337-47a7-b507-7076286d9963-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd54fa0-a337-47a7-b507-7076286d9963-istio-podinfo\") pod \"abd54fa0-a337-47a7-b507-7076286d9963\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") "
Oct 29 23:44:28.105260 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.105242    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd54fa0-a337-47a7-b507-7076286d9963-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd54fa0-a337-47a7-b507-7076286d9963-istiod-ca-cert\") pod \"abd54fa0-a337-47a7-b507-7076286d9963\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") "
Oct 29 23:44:28.105442 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.105425    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-istio-envoy\") pod \"abd54fa0-a337-47a7-b507-7076286d9963\" (UID: \"abd54fa0-a337-47a7-b507-7076286d9963\") "
Oct 29 23:44:28.106414 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.106388    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:28.106547 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.106532    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:28.106893 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.105067    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-workload-certs" pod "abd54fa0-a337-47a7-b507-7076286d9963" (UID: "abd54fa0-a337-47a7-b507-7076286d9963"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:44:28.107020 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.105228    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-credential-socket" pod "abd54fa0-a337-47a7-b507-7076286d9963" (UID: "abd54fa0-a337-47a7-b507-7076286d9963"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:44:28.107119 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.105769    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd54fa0-a337-47a7-b507-7076286d9963-istiod-ca-cert" pod "abd54fa0-a337-47a7-b507-7076286d9963" (UID: "abd54fa0-a337-47a7-b507-7076286d9963"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:44:28.109351 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.109327    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd54fa0-a337-47a7-b507-7076286d9963-istio-token" pod "abd54fa0-a337-47a7-b507-7076286d9963" (UID: "abd54fa0-a337-47a7-b507-7076286d9963"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:44:28.110936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.110896    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd54fa0-a337-47a7-b507-7076286d9963-kube-api-access-2pkzc" pod "abd54fa0-a337-47a7-b507-7076286d9963" (UID: "abd54fa0-a337-47a7-b507-7076286d9963"). InnerVolumeSpecName "kube-api-access-2pkzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:44:28.111155 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.111136    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abd54fa0-a337-47a7-b507-7076286d9963-istio-podinfo" pod "abd54fa0-a337-47a7-b507-7076286d9963" (UID: "abd54fa0-a337-47a7-b507-7076286d9963"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:44:28.112964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.112941    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-istio-envoy" pod "abd54fa0-a337-47a7-b507-7076286d9963" (UID: "abd54fa0-a337-47a7-b507-7076286d9963"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:44:28.213835 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.213779    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:28.213835 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.213837    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:28.214076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.213853    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd54fa0-a337-47a7-b507-7076286d9963-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:28.214076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.213873    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd54fa0-a337-47a7-b507-7076286d9963-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:28.214076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.213887    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd54fa0-a337-47a7-b507-7076286d9963-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:28.214076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.213899    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd54fa0-a337-47a7-b507-7076286d9963-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:28.214076 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.213912    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pkzc\" (UniqueName: \"kubernetes.io/projected/abd54fa0-a337-47a7-b507-7076286d9963-kube-api-access-2pkzc\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:28.512573 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.512535    5428 generic.go:356] "Generic (PLEG): container finished" podID="abd54fa0-a337-47a7-b507-7076286d9963" containerID="6e032b1b732ec57224d54ee1a3f883cc45cc77f9e8d8332f27124a4c198f792e" exitCode=0
Oct 29 23:44:28.513964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.513729    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf" event={"ID":"abd54fa0-a337-47a7-b507-7076286d9963","Type":"ContainerDied","Data":"6e032b1b732ec57224d54ee1a3f883cc45cc77f9e8d8332f27124a4c198f792e"}
Oct 29 23:44:28.514145 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.514012    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf" event={"ID":"abd54fa0-a337-47a7-b507-7076286d9963","Type":"ContainerDied","Data":"be50d49fdbb496d4e0a82736fb56f2a6f84336877501740fcaa710ef0070e8d3"}
Oct 29 23:44:28.514145 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.514047    5428 scope.go:122] "RemoveContainer" containerID="6e032b1b732ec57224d54ee1a3f883cc45cc77f9e8d8332f27124a4c198f792e"
Oct 29 23:44:28.514320 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.513741    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"
Oct 29 23:44:28.528966 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.528948    5428 scope.go:122] "RemoveContainer" containerID="6e032b1b732ec57224d54ee1a3f883cc45cc77f9e8d8332f27124a4c198f792e"
Oct 29 23:44:28.529255 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:44:28.529234    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e032b1b732ec57224d54ee1a3f883cc45cc77f9e8d8332f27124a4c198f792e\": container with ID starting with 6e032b1b732ec57224d54ee1a3f883cc45cc77f9e8d8332f27124a4c198f792e not found: ID does not exist" containerID="6e032b1b732ec57224d54ee1a3f883cc45cc77f9e8d8332f27124a4c198f792e"
Oct 29 23:44:28.529325 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.529261    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e032b1b732ec57224d54ee1a3f883cc45cc77f9e8d8332f27124a4c198f792e"} err="failed to get container status \"6e032b1b732ec57224d54ee1a3f883cc45cc77f9e8d8332f27124a4c198f792e\": rpc error: code = NotFound desc = could not find container \"6e032b1b732ec57224d54ee1a3f883cc45cc77f9e8d8332f27124a4c198f792e\": container with ID starting with 6e032b1b732ec57224d54ee1a3f883cc45cc77f9e8d8332f27124a4c198f792e not found: ID does not exist"
Oct 29 23:44:28.582879 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.582852    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"]
Oct 29 23:44:28.594293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.594271    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5cd0f2-openshift-default-994959c59-6msqf"]
Oct 29 23:44:28.780945 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:28.780733    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd54fa0-a337-47a7-b507-7076286d9963" path="/var/lib/kubelet/pods/abd54fa0-a337-47a7-b507-7076286d9963/volumes"
Oct 29 23:44:29.143357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:29.143310    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:29.255234 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:29.255206    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:31.173478 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:31.173445    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:32.486454 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:32.486417    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"]
Oct 29 23:44:32.486996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:32.486641    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t" podUID="e4ba52cf-6aac-4495-810c-8ba495c72d70" containerName="maas-api" containerID="cri-o://88fffa18ec8f06b605a8159f05fe565fd692960761daf8c09a89d70d61a884a4" gracePeriod=30
Oct 29 23:44:32.562177 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:32.562137    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:32.814452 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:32.814422    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"
Oct 29 23:44:32.854738 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:32.854701    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:32.915554 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:32.915519    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4ba52cf-6aac-4495-810c-8ba495c72d70-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4ba52cf-6aac-4495-810c-8ba495c72d70-maas-api-tls\") pod \"e4ba52cf-6aac-4495-810c-8ba495c72d70\" (UID: \"e4ba52cf-6aac-4495-810c-8ba495c72d70\") "
Oct 29 23:44:32.915758 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:32.915593    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ba52cf-6aac-4495-810c-8ba495c72d70-kube-api-access-h8kzm\" (UniqueName: \"kubernetes.io/projected/e4ba52cf-6aac-4495-810c-8ba495c72d70-kube-api-access-h8kzm\") pod \"e4ba52cf-6aac-4495-810c-8ba495c72d70\" (UID: \"e4ba52cf-6aac-4495-810c-8ba495c72d70\") "
Oct 29 23:44:32.915872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:32.915772    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4ba52cf-6aac-4495-810c-8ba495c72d70-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ba52cf-6aac-4495-810c-8ba495c72d70-metrics-certs\") pod \"e4ba52cf-6aac-4495-810c-8ba495c72d70\" (UID: \"e4ba52cf-6aac-4495-810c-8ba495c72d70\") "
Oct 29 23:44:32.916005 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:32.915907    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4ba52cf-6aac-4495-810c-8ba495c72d70-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ba52cf-6aac-4495-810c-8ba495c72d70-openshift-service-ca\") pod \"e4ba52cf-6aac-4495-810c-8ba495c72d70\" (UID: \"e4ba52cf-6aac-4495-810c-8ba495c72d70\") "
Oct 29 23:44:32.918860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:32.916587    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ba52cf-6aac-4495-810c-8ba495c72d70-openshift-service-ca" pod "e4ba52cf-6aac-4495-810c-8ba495c72d70" (UID: "e4ba52cf-6aac-4495-810c-8ba495c72d70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:44:32.919038 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:32.919013    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ba52cf-6aac-4495-810c-8ba495c72d70-maas-api-tls" pod "e4ba52cf-6aac-4495-810c-8ba495c72d70" (UID: "e4ba52cf-6aac-4495-810c-8ba495c72d70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:44:32.920201 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:32.920173    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ba52cf-6aac-4495-810c-8ba495c72d70-metrics-certs" pod "e4ba52cf-6aac-4495-810c-8ba495c72d70" (UID: "e4ba52cf-6aac-4495-810c-8ba495c72d70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:44:32.920351 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:32.920326    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ba52cf-6aac-4495-810c-8ba495c72d70-kube-api-access-h8kzm" pod "e4ba52cf-6aac-4495-810c-8ba495c72d70" (UID: "e4ba52cf-6aac-4495-810c-8ba495c72d70"). InnerVolumeSpecName "kube-api-access-h8kzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:44:33.019825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:33.019124    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ba52cf-6aac-4495-810c-8ba495c72d70-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:33.019825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:33.019168    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ba52cf-6aac-4495-810c-8ba495c72d70-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:33.019825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:33.019184    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4ba52cf-6aac-4495-810c-8ba495c72d70-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:33.019825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:33.019198    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8kzm\" (UniqueName: \"kubernetes.io/projected/e4ba52cf-6aac-4495-810c-8ba495c72d70-kube-api-access-h8kzm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:33.358314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:33.358285    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:33.541402 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:33.541368    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:33.564044 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:33.564018    5428 generic.go:356] "Generic (PLEG): container finished" podID="e4ba52cf-6aac-4495-810c-8ba495c72d70" containerID="88fffa18ec8f06b605a8159f05fe565fd692960761daf8c09a89d70d61a884a4" exitCode=0
Oct 29 23:44:33.564195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:33.564062    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"
Oct 29 23:44:33.564195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:33.564100    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t" event={"ID":"e4ba52cf-6aac-4495-810c-8ba495c72d70","Type":"ContainerDied","Data":"88fffa18ec8f06b605a8159f05fe565fd692960761daf8c09a89d70d61a884a4"}
Oct 29 23:44:33.564195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:33.564154    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t" event={"ID":"e4ba52cf-6aac-4495-810c-8ba495c72d70","Type":"ContainerDied","Data":"51c398a0185e5de403e55fe161b0ec1becfbddd94798647bd039b9ce2f125f46"}
Oct 29 23:44:33.564195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:33.564179    5428 scope.go:122] "RemoveContainer" containerID="88fffa18ec8f06b605a8159f05fe565fd692960761daf8c09a89d70d61a884a4"
Oct 29 23:44:33.577217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:33.577199    5428 scope.go:122] "RemoveContainer" containerID="88fffa18ec8f06b605a8159f05fe565fd692960761daf8c09a89d70d61a884a4"
Oct 29 23:44:33.577484 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:44:33.577465    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88fffa18ec8f06b605a8159f05fe565fd692960761daf8c09a89d70d61a884a4\": container with ID starting with 88fffa18ec8f06b605a8159f05fe565fd692960761daf8c09a89d70d61a884a4 not found: ID does not exist" containerID="88fffa18ec8f06b605a8159f05fe565fd692960761daf8c09a89d70d61a884a4"
Oct 29 23:44:33.577561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:33.577490    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88fffa18ec8f06b605a8159f05fe565fd692960761daf8c09a89d70d61a884a4"} err="failed to get container status \"88fffa18ec8f06b605a8159f05fe565fd692960761daf8c09a89d70d61a884a4\": rpc error: code = NotFound desc = could not find container \"88fffa18ec8f06b605a8159f05fe565fd692960761daf8c09a89d70d61a884a4\": container with ID starting with 88fffa18ec8f06b605a8159f05fe565fd692960761daf8c09a89d70d61a884a4 not found: ID does not exist"
Oct 29 23:44:33.626758 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:33.626725    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"]
Oct 29 23:44:33.637888 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:33.637860    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5e482-5f7fc6fb5-w2x9t"]
Oct 29 23:44:33.658721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:33.658696    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:33.766993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:33.766964    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:34.061752 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:34.061675    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:34.178603 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:34.178572    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:34.781241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:34.781205    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ba52cf-6aac-4495-810c-8ba495c72d70" path="/var/lib/kubelet/pods/e4ba52cf-6aac-4495-810c-8ba495c72d70/volumes"
Oct 29 23:44:35.571683 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:35.571647    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"]
Oct 29 23:44:35.571936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:35.571910    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww" podUID="d2b09df9-9a85-4fca-b1db-e17772dd738d" containerName="maas-api" containerID="cri-o://c458b5f7a42e8bfa9caa38e8fa3b4d2d1750a5565676b04bc3284a4f884836fb" gracePeriod=30
Oct 29 23:44:35.652428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:35.652378    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:35.893038 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:35.893001    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:44:35.986986 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:35.986957    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b09df9-9a85-4fca-b1db-e17772dd738d-kube-api-access-kjftp\" (UniqueName: \"kubernetes.io/projected/d2b09df9-9a85-4fca-b1db-e17772dd738d-kube-api-access-kjftp\") pod \"d2b09df9-9a85-4fca-b1db-e17772dd738d\" (UID: \"d2b09df9-9a85-4fca-b1db-e17772dd738d\") "
Oct 29 23:44:35.987180 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:35.987043    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b09df9-9a85-4fca-b1db-e17772dd738d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b09df9-9a85-4fca-b1db-e17772dd738d-openshift-service-ca\") pod \"d2b09df9-9a85-4fca-b1db-e17772dd738d\" (UID: \"d2b09df9-9a85-4fca-b1db-e17772dd738d\") "
Oct 29 23:44:35.987180 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:35.987090    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2b09df9-9a85-4fca-b1db-e17772dd738d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2b09df9-9a85-4fca-b1db-e17772dd738d-metrics-certs\") pod \"d2b09df9-9a85-4fca-b1db-e17772dd738d\" (UID: \"d2b09df9-9a85-4fca-b1db-e17772dd738d\") "
Oct 29 23:44:35.987180 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:35.987166    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2b09df9-9a85-4fca-b1db-e17772dd738d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b09df9-9a85-4fca-b1db-e17772dd738d-maas-api-tls\") pod \"d2b09df9-9a85-4fca-b1db-e17772dd738d\" (UID: \"d2b09df9-9a85-4fca-b1db-e17772dd738d\") "
Oct 29 23:44:35.987497 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:35.987464    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b09df9-9a85-4fca-b1db-e17772dd738d-openshift-service-ca" pod "d2b09df9-9a85-4fca-b1db-e17772dd738d" (UID: "d2b09df9-9a85-4fca-b1db-e17772dd738d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:44:35.989899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:35.989857    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b09df9-9a85-4fca-b1db-e17772dd738d-metrics-certs" pod "d2b09df9-9a85-4fca-b1db-e17772dd738d" (UID: "d2b09df9-9a85-4fca-b1db-e17772dd738d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:44:35.990265 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:35.990221    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b09df9-9a85-4fca-b1db-e17772dd738d-maas-api-tls" pod "d2b09df9-9a85-4fca-b1db-e17772dd738d" (UID: "d2b09df9-9a85-4fca-b1db-e17772dd738d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:44:35.990422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:35.990397    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b09df9-9a85-4fca-b1db-e17772dd738d-kube-api-access-kjftp" pod "d2b09df9-9a85-4fca-b1db-e17772dd738d" (UID: "d2b09df9-9a85-4fca-b1db-e17772dd738d"). InnerVolumeSpecName "kube-api-access-kjftp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:44:35.991343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:35.991317    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjftp\" (UniqueName: \"kubernetes.io/projected/d2b09df9-9a85-4fca-b1db-e17772dd738d-kube-api-access-kjftp\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:35.991491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:35.991346    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b09df9-9a85-4fca-b1db-e17772dd738d-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:35.991491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:35.991362    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2b09df9-9a85-4fca-b1db-e17772dd738d-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:35.991491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:35.991376    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b09df9-9a85-4fca-b1db-e17772dd738d-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:36.345781 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:36.345751    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:36.594954 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:36.594897    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:36.635040 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:36.634998    5428 generic.go:356] "Generic (PLEG): container finished" podID="d2b09df9-9a85-4fca-b1db-e17772dd738d" containerID="c458b5f7a42e8bfa9caa38e8fa3b4d2d1750a5565676b04bc3284a4f884836fb" exitCode=0
Oct 29 23:44:36.635235 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:36.635114    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"
Oct 29 23:44:36.635235 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:36.635145    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww" event={"ID":"d2b09df9-9a85-4fca-b1db-e17772dd738d","Type":"ContainerDied","Data":"c458b5f7a42e8bfa9caa38e8fa3b4d2d1750a5565676b04bc3284a4f884836fb"}
Oct 29 23:44:36.635235 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:36.635201    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww" event={"ID":"d2b09df9-9a85-4fca-b1db-e17772dd738d","Type":"ContainerDied","Data":"c24423670cd560c38e2ec441e58079cc1f31d910eb7b0779b07ff27a1bf5dcca"}
Oct 29 23:44:36.635235 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:36.635228    5428 scope.go:122] "RemoveContainer" containerID="c458b5f7a42e8bfa9caa38e8fa3b4d2d1750a5565676b04bc3284a4f884836fb"
Oct 29 23:44:36.652779 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:36.652733    5428 scope.go:122] "RemoveContainer" containerID="c458b5f7a42e8bfa9caa38e8fa3b4d2d1750a5565676b04bc3284a4f884836fb"
Oct 29 23:44:36.653550 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:44:36.653501    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c458b5f7a42e8bfa9caa38e8fa3b4d2d1750a5565676b04bc3284a4f884836fb\": container with ID starting with c458b5f7a42e8bfa9caa38e8fa3b4d2d1750a5565676b04bc3284a4f884836fb not found: ID does not exist" containerID="c458b5f7a42e8bfa9caa38e8fa3b4d2d1750a5565676b04bc3284a4f884836fb"
Oct 29 23:44:36.653550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:36.653562    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c458b5f7a42e8bfa9caa38e8fa3b4d2d1750a5565676b04bc3284a4f884836fb"} err="failed to get container status \"c458b5f7a42e8bfa9caa38e8fa3b4d2d1750a5565676b04bc3284a4f884836fb\": rpc error: code = NotFound desc = could not find container \"c458b5f7a42e8bfa9caa38e8fa3b4d2d1750a5565676b04bc3284a4f884836fb\": container with ID starting with c458b5f7a42e8bfa9caa38e8fa3b4d2d1750a5565676b04bc3284a4f884836fb not found: ID does not exist"
Oct 29 23:44:36.694685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:36.694654    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"]
Oct 29 23:44:36.699911 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:36.699885    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6d9e36-75c4d76c4d-r87ww"]
Oct 29 23:44:36.792936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:36.792903    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b09df9-9a85-4fca-b1db-e17772dd738d" path="/var/lib/kubelet/pods/d2b09df9-9a85-4fca-b1db-e17772dd738d/volumes"
Oct 29 23:44:36.901658 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:36.901582    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:37.202319 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:37.202234    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:38.775301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:38.775251    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:39.770615 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:39.768015    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:41.012087 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:41.012055    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:41.154343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:41.154303    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:41.368146 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:41.368105    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:43.086099 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:43.086051    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:43.388767 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:43.388731    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:43.514015 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:43.513984    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:43.734750 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:43.734662    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:45.188734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:45.188701    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:47.011660 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:47.011619    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:47.368648 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:44:47.368604    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:44:48.508642 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:48.508604    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9"]
Oct 29 23:44:48.654438 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:48.654399    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:48.770109 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:48.770006    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:50.486318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:50.486287    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:52.206858 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:52.206789    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:52.418925 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:52.418894    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"]
Oct 29 23:44:52.451166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:52.451136    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"]
Oct 29 23:44:52.568628 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:52.568545    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:52.694789 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:52.694724    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:53.509294 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:53.509252    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9" podUID="df3df4b7-f563-47b9-8be3-62f09de6d5fb" containerName="payload-processing" containerID="cri-o://73fb9dd75b3db4a49e404688c01cca8f4b3dc20f427f107dd8c07c2b674c6806" gracePeriod=25
Oct 29 23:44:53.800015 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:53.799977    5428 generic.go:356] "Generic (PLEG): container finished" podID="df3df4b7-f563-47b9-8be3-62f09de6d5fb" containerID="73fb9dd75b3db4a49e404688c01cca8f4b3dc20f427f107dd8c07c2b674c6806" exitCode=0
Oct 29 23:44:53.800361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:53.800338    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9" event={"ID":"df3df4b7-f563-47b9-8be3-62f09de6d5fb","Type":"ContainerDied","Data":"73fb9dd75b3db4a49e404688c01cca8f4b3dc20f427f107dd8c07c2b674c6806"}
Oct 29 23:44:53.800524 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:53.800507    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9" event={"ID":"df3df4b7-f563-47b9-8be3-62f09de6d5fb","Type":"ContainerDied","Data":"35e2e6bb605a91778e3b5358d343442059031d5f50d4cc39015da9edef1ffcac"}
Oct 29 23:44:53.800638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:53.800625    5428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35e2e6bb605a91778e3b5358d343442059031d5f50d4cc39015da9edef1ffcac"
Oct 29 23:44:53.834823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:53.828900    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9"
Oct 29 23:44:53.957366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:53.957334    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df3df4b7-f563-47b9-8be3-62f09de6d5fb-kube-api-access-chckp\" (UniqueName: \"kubernetes.io/projected/df3df4b7-f563-47b9-8be3-62f09de6d5fb-kube-api-access-chckp\") pod \"df3df4b7-f563-47b9-8be3-62f09de6d5fb\" (UID: \"df3df4b7-f563-47b9-8be3-62f09de6d5fb\") "
Oct 29 23:44:53.957552 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:53.957466    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df3df4b7-f563-47b9-8be3-62f09de6d5fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3df4b7-f563-47b9-8be3-62f09de6d5fb-plugins-config-volume\") pod \"df3df4b7-f563-47b9-8be3-62f09de6d5fb\" (UID: \"df3df4b7-f563-47b9-8be3-62f09de6d5fb\") "
Oct 29 23:44:53.958936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:53.958909    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df3df4b7-f563-47b9-8be3-62f09de6d5fb-plugins-config-volume" pod "df3df4b7-f563-47b9-8be3-62f09de6d5fb" (UID: "df3df4b7-f563-47b9-8be3-62f09de6d5fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:44:53.963124 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:53.963088    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3df4b7-f563-47b9-8be3-62f09de6d5fb-kube-api-access-chckp" pod "df3df4b7-f563-47b9-8be3-62f09de6d5fb" (UID: "df3df4b7-f563-47b9-8be3-62f09de6d5fb"). InnerVolumeSpecName "kube-api-access-chckp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:44:54.060002 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:54.059918    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3df4b7-f563-47b9-8be3-62f09de6d5fb-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:54.060002 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:54.059944    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chckp\" (UniqueName: \"kubernetes.io/projected/df3df4b7-f563-47b9-8be3-62f09de6d5fb-kube-api-access-chckp\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:44:54.391412 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:54.391382    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:54.588161 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:54.588124    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br"]
Oct 29 23:44:54.623193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:54.623159    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m"]
Oct 29 23:44:54.724670 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:54.724587    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:54.816290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:54.816256    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9"
Oct 29 23:44:54.849512 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:54.849483    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:54.870772 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:54.870720    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9"]
Oct 29 23:44:54.877387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:54.877362    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55c3fc-789956868d-wnhw9"]
Oct 29 23:44:55.924518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:55.924486    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:56.515331 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:56.515295    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:56.800895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:56.800792    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df3df4b7-f563-47b9-8be3-62f09de6d5fb" path="/var/lib/kubelet/pods/df3df4b7-f563-47b9-8be3-62f09de6d5fb/volumes"
Oct 29 23:44:57.419769 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:57.419707    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj" podUID="9f1310de-26bc-46b9-86d6-10c8d1d49a93" containerName="payload-processing" containerID="cri-o://cbe4b6348319836c795de7c18c0186ebf073f26d8ab851f7fb7a09cb8a776c82" gracePeriod=25
Oct 29 23:44:57.452154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:57.452117    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd" podUID="9553cf74-cf03-4ad2-8241-d82087a823f9" containerName="payload-pre-processing" containerID="cri-o://9473cc06c02a0b13020b16e281c4f2215afbf64e24a22ab3e9c07263362db820" gracePeriod=25
Oct 29 23:44:58.018814 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:58.018767    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd" podUID="9553cf74-cf03-4ad2-8241-d82087a823f9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:44:58.018995 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:58.018797    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj" podUID="9f1310de-26bc-46b9-86d6-10c8d1d49a93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:44:58.294273 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:58.294171    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:58.496971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:58.496937    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:58.620565 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:58.620533    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:44:59.588902 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:59.588863    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br" podUID="7850180b-94dc-4e7b-9323-b2ba77d1884d" containerName="payload-processing" containerID="cri-o://45f9a74cc7fc6d2d7ee2ef8f9777dc104219d995d5878cfd28ee841542cacc34" gracePeriod=25
Oct 29 23:44:59.624207 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:44:59.624176    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m" podUID="984a1e35-8988-450a-80ad-76b3bfe60f38" containerName="payload-pre-processing" containerID="cri-o://29f1b54421724b2a054c1c763dfbc1aa6546cec1ae05f8c3a1f3dfee3e0137f2" gracePeriod=25
Oct 29 23:45:00.009103 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.009068    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:00.276539 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.276457    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n"]
Oct 29 23:45:00.277873 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.277830    5428 cpu_manager.go:414] "containerMap: removing container" podUID="d2b09df9-9a85-4fca-b1db-e17772dd738d" containerName="maas-api"
Oct 29 23:45:00.277873 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.277854    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b09df9-9a85-4fca-b1db-e17772dd738d" containerName="maas-api"
Oct 29 23:45:00.278068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.277879    5428 cpu_manager.go:414] "containerMap: removing container" podUID="abd54fa0-a337-47a7-b507-7076286d9963" containerName="istio-proxy"
Oct 29 23:45:00.278068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.277888    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd54fa0-a337-47a7-b507-7076286d9963" containerName="istio-proxy"
Oct 29 23:45:00.278068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.277924    5428 cpu_manager.go:414] "containerMap: removing container" podUID="df3df4b7-f563-47b9-8be3-62f09de6d5fb" containerName="payload-processing"
Oct 29 23:45:00.278068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.277931    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df3df4b7-f563-47b9-8be3-62f09de6d5fb" containerName="payload-processing"
Oct 29 23:45:00.278068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.277952    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e4ba52cf-6aac-4495-810c-8ba495c72d70" containerName="maas-api"
Oct 29 23:45:00.278068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.277959    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ba52cf-6aac-4495-810c-8ba495c72d70" containerName="maas-api"
Oct 29 23:45:00.278391 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.278168    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b09df9-9a85-4fca-b1db-e17772dd738d" containerName="maas-api"
Oct 29 23:45:00.278391 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.278181    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd54fa0-a337-47a7-b507-7076286d9963" containerName="istio-proxy"
Oct 29 23:45:00.278391 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.278192    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="df3df4b7-f563-47b9-8be3-62f09de6d5fb" containerName="payload-processing"
Oct 29 23:45:00.278391 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.278199    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ba52cf-6aac-4495-810c-8ba495c72d70" containerName="maas-api"
Oct 29 23:45:00.284006 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.283985    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n"
Oct 29 23:45:00.300932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.300909    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75"]
Oct 29 23:45:00.308103 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.308079    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75"
Oct 29 23:45:00.314957 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.314928    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n"]
Oct 29 23:45:00.331436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.331411    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727"]
Oct 29 23:45:00.343763 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.342277    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75"]
Oct 29 23:45:00.343763 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.342384    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727"
Oct 29 23:45:00.346129 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.346099    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4"]
Oct 29 23:45:00.356333 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.356300    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4"
Oct 29 23:45:00.362558 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.362530    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mh6d\" (UniqueName: \"kubernetes.io/projected/3ba2006e-ff86-47a3-8d81-c935530476e8-kube-api-access-5mh6d\") pod \"maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n\" (UID: \"3ba2006e-ff86-47a3-8d81-c935530476e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n"
Oct 29 23:45:00.362681 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.362576    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctr7j\" (UniqueName: \"kubernetes.io/projected/918c3198-f2b2-413c-9002-91b324b0aeae-kube-api-access-ctr7j\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75\" (UID: \"918c3198-f2b2-413c-9002-91b324b0aeae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75"
Oct 29 23:45:00.362762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.362697    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ba2006e-ff86-47a3-8d81-c935530476e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n\" (UID: \"3ba2006e-ff86-47a3-8d81-c935530476e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n"
Oct 29 23:45:00.362925 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.362907    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd5qh\" (UniqueName: \"kubernetes.io/projected/b0a1fa0e-f264-46bb-83c0-efd1be480dd3-kube-api-access-sd5qh\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727\" (UID: \"b0a1fa0e-f264-46bb-83c0-efd1be480dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727"
Oct 29 23:45:00.363038 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.362977    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lncc\" (UniqueName: \"kubernetes.io/projected/4ff49c0b-5f9d-411a-a767-77897583a7a3-kube-api-access-7lncc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4\" (UID: \"4ff49c0b-5f9d-411a-a767-77897583a7a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4"
Oct 29 23:45:00.363102 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.363033    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/918c3198-f2b2-413c-9002-91b324b0aeae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75\" (UID: \"918c3198-f2b2-413c-9002-91b324b0aeae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75"
Oct 29 23:45:00.363161 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.363137    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff49c0b-5f9d-411a-a767-77897583a7a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4\" (UID: \"4ff49c0b-5f9d-411a-a767-77897583a7a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4"
Oct 29 23:45:00.363223 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.363177    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a1fa0e-f264-46bb-83c0-efd1be480dd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727\" (UID: \"b0a1fa0e-f264-46bb-83c0-efd1be480dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727"
Oct 29 23:45:00.364363 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.364343    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm"]
Oct 29 23:45:00.372495 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.371895    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\""
Oct 29 23:45:00.377990 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.377965    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727"]
Oct 29 23:45:00.378256 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.378224    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm"
Oct 29 23:45:00.403762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.403728    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4"]
Oct 29 23:45:00.413235 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.411228    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk"]
Oct 29 23:45:00.421012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.420988    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm"]
Oct 29 23:45:00.421150 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.421095    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk"
Oct 29 23:45:00.440857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.440751    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc"]
Oct 29 23:45:00.457730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.457705    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc"
Oct 29 23:45:00.467357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.465267    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lncc\" (UniqueName: \"kubernetes.io/projected/4ff49c0b-5f9d-411a-a767-77897583a7a3-kube-api-access-7lncc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4\" (UID: \"4ff49c0b-5f9d-411a-a767-77897583a7a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4"
Oct 29 23:45:00.467357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.465316    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2bzc\" (UniqueName: \"kubernetes.io/projected/fc6e9365-f21c-49b3-b513-c17c1113cb41-kube-api-access-j2bzc\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk\" (UID: \"fc6e9365-f21c-49b3-b513-c17c1113cb41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk"
Oct 29 23:45:00.467357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.465396    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8fcfb61-2011-477e-9138-c7eb186665a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm\" (UID: \"d8fcfb61-2011-477e-9138-c7eb186665a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm"
Oct 29 23:45:00.467357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.465431    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/918c3198-f2b2-413c-9002-91b324b0aeae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75\" (UID: \"918c3198-f2b2-413c-9002-91b324b0aeae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75"
Oct 29 23:45:00.467357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.465492    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc6e9365-f21c-49b3-b513-c17c1113cb41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk\" (UID: \"fc6e9365-f21c-49b3-b513-c17c1113cb41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk"
Oct 29 23:45:00.467357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.465568    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2npg\" (UniqueName: \"kubernetes.io/projected/d8fcfb61-2011-477e-9138-c7eb186665a6-kube-api-access-d2npg\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm\" (UID: \"d8fcfb61-2011-477e-9138-c7eb186665a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm"
Oct 29 23:45:00.467357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.465606    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff49c0b-5f9d-411a-a767-77897583a7a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4\" (UID: \"4ff49c0b-5f9d-411a-a767-77897583a7a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4"
Oct 29 23:45:00.467357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.465648    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a1fa0e-f264-46bb-83c0-efd1be480dd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727\" (UID: \"b0a1fa0e-f264-46bb-83c0-efd1be480dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727"
Oct 29 23:45:00.467357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.465754    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffbf666-ed06-4d62-bf9a-d1786bcf071f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc\" (UID: \"9ffbf666-ed06-4d62-bf9a-d1786bcf071f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc"
Oct 29 23:45:00.467357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.465867    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mh6d\" (UniqueName: \"kubernetes.io/projected/3ba2006e-ff86-47a3-8d81-c935530476e8-kube-api-access-5mh6d\") pod \"maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n\" (UID: \"3ba2006e-ff86-47a3-8d81-c935530476e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n"
Oct 29 23:45:00.467357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.465900    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctr7j\" (UniqueName: \"kubernetes.io/projected/918c3198-f2b2-413c-9002-91b324b0aeae-kube-api-access-ctr7j\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75\" (UID: \"918c3198-f2b2-413c-9002-91b324b0aeae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75"
Oct 29 23:45:00.467357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.465935    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ba2006e-ff86-47a3-8d81-c935530476e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n\" (UID: \"3ba2006e-ff86-47a3-8d81-c935530476e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n"
Oct 29 23:45:00.467357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.466002    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t4qh\" (UniqueName: \"kubernetes.io/projected/9ffbf666-ed06-4d62-bf9a-d1786bcf071f-kube-api-access-7t4qh\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc\" (UID: \"9ffbf666-ed06-4d62-bf9a-d1786bcf071f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc"
Oct 29 23:45:00.467357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.466097    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd5qh\" (UniqueName: \"kubernetes.io/projected/b0a1fa0e-f264-46bb-83c0-efd1be480dd3-kube-api-access-sd5qh\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727\" (UID: \"b0a1fa0e-f264-46bb-83c0-efd1be480dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727"
Oct 29 23:45:00.467357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.466325    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/918c3198-f2b2-413c-9002-91b324b0aeae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75\" (UID: \"918c3198-f2b2-413c-9002-91b324b0aeae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75"
Oct 29 23:45:00.467357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.467115    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a1fa0e-f264-46bb-83c0-efd1be480dd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727\" (UID: \"b0a1fa0e-f264-46bb-83c0-efd1be480dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727"
Oct 29 23:45:00.467357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.467161    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn"]
Oct 29 23:45:00.468313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.467453    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ba2006e-ff86-47a3-8d81-c935530476e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n\" (UID: \"3ba2006e-ff86-47a3-8d81-c935530476e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n"
Oct 29 23:45:00.468313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.468120    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff49c0b-5f9d-411a-a767-77897583a7a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4\" (UID: \"4ff49c0b-5f9d-411a-a767-77897583a7a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4"
Oct 29 23:45:00.476658 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.476633    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk"]
Oct 29 23:45:00.476951 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.476930    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn"
Oct 29 23:45:00.481890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.481871    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc"]
Oct 29 23:45:00.489259 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.489122    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn"]
Oct 29 23:45:00.493638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.493608    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mh6d\" (UniqueName: \"kubernetes.io/projected/3ba2006e-ff86-47a3-8d81-c935530476e8-kube-api-access-5mh6d\") pod \"maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n\" (UID: \"3ba2006e-ff86-47a3-8d81-c935530476e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n"
Oct 29 23:45:00.494498 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.494469    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctr7j\" (UniqueName: \"kubernetes.io/projected/918c3198-f2b2-413c-9002-91b324b0aeae-kube-api-access-ctr7j\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75\" (UID: \"918c3198-f2b2-413c-9002-91b324b0aeae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75"
Oct 29 23:45:00.494598 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.494472    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd5qh\" (UniqueName: \"kubernetes.io/projected/b0a1fa0e-f264-46bb-83c0-efd1be480dd3-kube-api-access-sd5qh\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727\" (UID: \"b0a1fa0e-f264-46bb-83c0-efd1be480dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727"
Oct 29 23:45:00.495305 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.495283    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lncc\" (UniqueName: \"kubernetes.io/projected/4ff49c0b-5f9d-411a-a767-77897583a7a3-kube-api-access-7lncc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4\" (UID: \"4ff49c0b-5f9d-411a-a767-77897583a7a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4"
Oct 29 23:45:00.567030 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.566943    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t4qh\" (UniqueName: \"kubernetes.io/projected/9ffbf666-ed06-4d62-bf9a-d1786bcf071f-kube-api-access-7t4qh\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc\" (UID: \"9ffbf666-ed06-4d62-bf9a-d1786bcf071f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc"
Oct 29 23:45:00.567030 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.567001    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d961cc-02de-4c2d-8f84-87e66cf01259-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn\" (UID: \"36d961cc-02de-4c2d-8f84-87e66cf01259\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn"
Oct 29 23:45:00.567233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.567128    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2bzc\" (UniqueName: \"kubernetes.io/projected/fc6e9365-f21c-49b3-b513-c17c1113cb41-kube-api-access-j2bzc\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk\" (UID: \"fc6e9365-f21c-49b3-b513-c17c1113cb41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk"
Oct 29 23:45:00.567233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.567174    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8fcfb61-2011-477e-9138-c7eb186665a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm\" (UID: \"d8fcfb61-2011-477e-9138-c7eb186665a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm"
Oct 29 23:45:00.567233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.567213    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc6e9365-f21c-49b3-b513-c17c1113cb41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk\" (UID: \"fc6e9365-f21c-49b3-b513-c17c1113cb41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk"
Oct 29 23:45:00.567396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.567260    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2npg\" (UniqueName: \"kubernetes.io/projected/d8fcfb61-2011-477e-9138-c7eb186665a6-kube-api-access-d2npg\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm\" (UID: \"d8fcfb61-2011-477e-9138-c7eb186665a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm"
Oct 29 23:45:00.567396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.567290    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtcrt\" (UniqueName: \"kubernetes.io/projected/36d961cc-02de-4c2d-8f84-87e66cf01259-kube-api-access-wtcrt\") pod \"maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn\" (UID: \"36d961cc-02de-4c2d-8f84-87e66cf01259\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn"
Oct 29 23:45:00.567396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.567345    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffbf666-ed06-4d62-bf9a-d1786bcf071f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc\" (UID: \"9ffbf666-ed06-4d62-bf9a-d1786bcf071f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc"
Oct 29 23:45:00.568045 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.568020    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8fcfb61-2011-477e-9138-c7eb186665a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm\" (UID: \"d8fcfb61-2011-477e-9138-c7eb186665a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm"
Oct 29 23:45:00.568153 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.568073    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc6e9365-f21c-49b3-b513-c17c1113cb41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk\" (UID: \"fc6e9365-f21c-49b3-b513-c17c1113cb41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk"
Oct 29 23:45:00.568153 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.568130    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffbf666-ed06-4d62-bf9a-d1786bcf071f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc\" (UID: \"9ffbf666-ed06-4d62-bf9a-d1786bcf071f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc"
Oct 29 23:45:00.589971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.589943    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t4qh\" (UniqueName: \"kubernetes.io/projected/9ffbf666-ed06-4d62-bf9a-d1786bcf071f-kube-api-access-7t4qh\") pod \"maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc\" (UID: \"9ffbf666-ed06-4d62-bf9a-d1786bcf071f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc"
Oct 29 23:45:00.590781 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.590761    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2bzc\" (UniqueName: \"kubernetes.io/projected/fc6e9365-f21c-49b3-b513-c17c1113cb41-kube-api-access-j2bzc\") pod \"maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk\" (UID: \"fc6e9365-f21c-49b3-b513-c17c1113cb41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk"
Oct 29 23:45:00.591044 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.591024    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2npg\" (UniqueName: \"kubernetes.io/projected/d8fcfb61-2011-477e-9138-c7eb186665a6-kube-api-access-d2npg\") pod \"maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm\" (UID: \"d8fcfb61-2011-477e-9138-c7eb186665a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm"
Oct 29 23:45:00.615474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.615449    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n"
Oct 29 23:45:00.628397 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.628372    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75"
Oct 29 23:45:00.668962 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.668846    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtcrt\" (UniqueName: \"kubernetes.io/projected/36d961cc-02de-4c2d-8f84-87e66cf01259-kube-api-access-wtcrt\") pod \"maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn\" (UID: \"36d961cc-02de-4c2d-8f84-87e66cf01259\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn"
Oct 29 23:45:00.669188 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.669049    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d961cc-02de-4c2d-8f84-87e66cf01259-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn\" (UID: \"36d961cc-02de-4c2d-8f84-87e66cf01259\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn"
Oct 29 23:45:00.669710 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.669688    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d961cc-02de-4c2d-8f84-87e66cf01259-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn\" (UID: \"36d961cc-02de-4c2d-8f84-87e66cf01259\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn"
Oct 29 23:45:00.684205 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.684022    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727"
Oct 29 23:45:00.702610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.702468    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4"
Oct 29 23:45:00.712558 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.712498    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtcrt\" (UniqueName: \"kubernetes.io/projected/36d961cc-02de-4c2d-8f84-87e66cf01259-kube-api-access-wtcrt\") pod \"maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn\" (UID: \"36d961cc-02de-4c2d-8f84-87e66cf01259\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn"
Oct 29 23:45:00.721480 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.721264    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm"
Oct 29 23:45:00.766864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.762115    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk"
Oct 29 23:45:00.786514 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.786483    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc"
Oct 29 23:45:00.802928 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:00.799092    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn"
Oct 29 23:45:01.056247 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.056221    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727"]
Oct 29 23:45:01.058263 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:45:01.058230    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a1fa0e_f264_46bb_83c0_efd1be480dd3.slice/crio-970ae0cc3b75927e1e47667cac259f996b8054fb33d1f5dd2e071c23881909bc WatchSource:0}: Error finding container 970ae0cc3b75927e1e47667cac259f996b8054fb33d1f5dd2e071c23881909bc: Status 404 returned error can't find the container with id 970ae0cc3b75927e1e47667cac259f996b8054fb33d1f5dd2e071c23881909bc
Oct 29 23:45:01.072382 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.072354    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4"]
Oct 29 23:45:01.078044 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:45:01.077994    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff49c0b_5f9d_411a_a767_77897583a7a3.slice/crio-0ceb70d32f9f496c56c8e8f5d2a982bb2c2047b25658406c5f71367f12397373 WatchSource:0}: Error finding container 0ceb70d32f9f496c56c8e8f5d2a982bb2c2047b25658406c5f71367f12397373: Status 404 returned error can't find the container with id 0ceb70d32f9f496c56c8e8f5d2a982bb2c2047b25658406c5f71367f12397373
Oct 29 23:45:01.090903 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:45:01.090836    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8fcfb61_2011_477e_9138_c7eb186665a6.slice/crio-2f7f6fae4bd862227f48bbc277d8087b44105a85c3d73512d6301544779d8c4d WatchSource:0}: Error finding container 2f7f6fae4bd862227f48bbc277d8087b44105a85c3d73512d6301544779d8c4d: Status 404 returned error can't find the container with id 2f7f6fae4bd862227f48bbc277d8087b44105a85c3d73512d6301544779d8c4d
Oct 29 23:45:01.095452 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.095427    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm"]
Oct 29 23:45:01.134463 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:45:01.134425    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6e9365_f21c_49b3_b513_c17c1113cb41.slice/crio-bce97047fd5a07e3973a6db0b8c3ea7b0261d7089bfed0d73ce268225e606116 WatchSource:0}: Error finding container bce97047fd5a07e3973a6db0b8c3ea7b0261d7089bfed0d73ce268225e606116: Status 404 returned error can't find the container with id bce97047fd5a07e3973a6db0b8c3ea7b0261d7089bfed0d73ce268225e606116
Oct 29 23:45:01.135548 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.135526    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk"]
Oct 29 23:45:01.137743 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:45:01.137714    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba2006e_ff86_47a3_8d81_c935530476e8.slice/crio-2104a31b364b9c455b8d381d37f81930f78b38358f4495921a074e704b409a64 WatchSource:0}: Error finding container 2104a31b364b9c455b8d381d37f81930f78b38358f4495921a074e704b409a64: Status 404 returned error can't find the container with id 2104a31b364b9c455b8d381d37f81930f78b38358f4495921a074e704b409a64
Oct 29 23:45:01.142177 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.142157    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n"]
Oct 29 23:45:01.149494 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.149469    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75"]
Oct 29 23:45:01.151893 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:45:01.151863    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918c3198_f2b2_413c_9002_91b324b0aeae.slice/crio-2a53ece70a2f498e32f99bf0f9087a210615ac91165b10b872bdcbf7208ca50d WatchSource:0}: Error finding container 2a53ece70a2f498e32f99bf0f9087a210615ac91165b10b872bdcbf7208ca50d: Status 404 returned error can't find the container with id 2a53ece70a2f498e32f99bf0f9087a210615ac91165b10b872bdcbf7208ca50d
Oct 29 23:45:01.177785 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.177756    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc"]
Oct 29 23:45:01.186133 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.186110    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn"]
Oct 29 23:45:01.277999 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:45:01.277970    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d961cc_02de_4c2d_8f84_87e66cf01259.slice/crio-d2d04900eeab1832c55f55698b44f3ef1db9b2946b18a82e24abd761fab4b594 WatchSource:0}: Error finding container d2d04900eeab1832c55f55698b44f3ef1db9b2946b18a82e24abd761fab4b594: Status 404 returned error can't find the container with id d2d04900eeab1832c55f55698b44f3ef1db9b2946b18a82e24abd761fab4b594
Oct 29 23:45:01.918092 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.918003    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm" event={"ID":"d8fcfb61-2011-477e-9138-c7eb186665a6","Type":"ContainerStarted","Data":"3ddbf4d1f397e2abc7c685c54643bf94ad9f0f30c028dc6f3563bf8f3482345a"}
Oct 29 23:45:01.918092 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.918061    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm" event={"ID":"d8fcfb61-2011-477e-9138-c7eb186665a6","Type":"ContainerStarted","Data":"2f7f6fae4bd862227f48bbc277d8087b44105a85c3d73512d6301544779d8c4d"}
Oct 29 23:45:01.920122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.920091    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n" event={"ID":"3ba2006e-ff86-47a3-8d81-c935530476e8","Type":"ContainerStarted","Data":"2cca1c7fcb29597c58baf82ed81d7e30c94d3bedacc61960c65e4fcbd71d0ad1"}
Oct 29 23:45:01.920248 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.920144    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n" event={"ID":"3ba2006e-ff86-47a3-8d81-c935530476e8","Type":"ContainerStarted","Data":"2104a31b364b9c455b8d381d37f81930f78b38358f4495921a074e704b409a64"}
Oct 29 23:45:01.921855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.921827    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4" event={"ID":"4ff49c0b-5f9d-411a-a767-77897583a7a3","Type":"ContainerStarted","Data":"5dd9390b739cc1870b92fc7c7efa4c4f32164b8b63f34d05b18b2de88d17f7b8"}
Oct 29 23:45:01.921960 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.921879    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4" event={"ID":"4ff49c0b-5f9d-411a-a767-77897583a7a3","Type":"ContainerStarted","Data":"0ceb70d32f9f496c56c8e8f5d2a982bb2c2047b25658406c5f71367f12397373"}
Oct 29 23:45:01.923512 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.923489    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn" event={"ID":"36d961cc-02de-4c2d-8f84-87e66cf01259","Type":"ContainerStarted","Data":"b9db57f6877b650ffbb9b63a72fded4500bc45a081fc0f886d80452e4c7fa520"}
Oct 29 23:45:01.923620 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.923536    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn" event={"ID":"36d961cc-02de-4c2d-8f84-87e66cf01259","Type":"ContainerStarted","Data":"d2d04900eeab1832c55f55698b44f3ef1db9b2946b18a82e24abd761fab4b594"}
Oct 29 23:45:01.924794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.924770    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk" event={"ID":"fc6e9365-f21c-49b3-b513-c17c1113cb41","Type":"ContainerStarted","Data":"bce97047fd5a07e3973a6db0b8c3ea7b0261d7089bfed0d73ce268225e606116"}
Oct 29 23:45:01.926223 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.926203    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc" event={"ID":"9ffbf666-ed06-4d62-bf9a-d1786bcf071f","Type":"ContainerStarted","Data":"cdb28d41e60cea6bb214d278b50758b820daca585dcc6f1d32cfb934dc41d7c9"}
Oct 29 23:45:01.927645 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.927623    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727" event={"ID":"b0a1fa0e-f264-46bb-83c0-efd1be480dd3","Type":"ContainerStarted","Data":"fc5a606e32e1b863e050c6f065f43658609728f5625e9a3249dbbdc52b891c7a"}
Oct 29 23:45:01.927732 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.927660    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727" event={"ID":"b0a1fa0e-f264-46bb-83c0-efd1be480dd3","Type":"ContainerStarted","Data":"970ae0cc3b75927e1e47667cac259f996b8054fb33d1f5dd2e071c23881909bc"}
Oct 29 23:45:01.929035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.929014    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75" event={"ID":"918c3198-f2b2-413c-9002-91b324b0aeae","Type":"ContainerStarted","Data":"2a53ece70a2f498e32f99bf0f9087a210615ac91165b10b872bdcbf7208ca50d"}
Oct 29 23:45:01.958959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.958922    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm" podStartSLOduration=1.958905797 podStartE2EDuration="1.958905797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:01.957411201 +0000 UTC m=+3705.300359947" watchObservedRunningTime="2026-10-29 23:45:01.958905797 +0000 UTC m=+3705.301854542"
Oct 29 23:45:01.999336 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:01.999293    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n" podStartSLOduration=1.999279657 podStartE2EDuration="1.999279657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:01.995830391 +0000 UTC m=+3705.338779136" watchObservedRunningTime="2026-10-29 23:45:01.999279657 +0000 UTC m=+3705.342228402"
Oct 29 23:45:02.038377 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.038329    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn" podStartSLOduration=2.038316335 podStartE2EDuration="2.038316335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:02.034729884 +0000 UTC m=+3705.377678629" watchObservedRunningTime="2026-10-29 23:45:02.038316335 +0000 UTC m=+3705.381265127"
Oct 29 23:45:02.075347 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.075301    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4" podStartSLOduration=2.075284025 podStartE2EDuration="2.075284025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:02.073340903 +0000 UTC m=+3705.416289652" watchObservedRunningTime="2026-10-29 23:45:02.075284025 +0000 UTC m=+3705.418232769"
Oct 29 23:45:02.111926 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.111881    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727" podStartSLOduration=2.111865103 podStartE2EDuration="2.111865103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:02.111483306 +0000 UTC m=+3705.454432052" watchObservedRunningTime="2026-10-29 23:45:02.111865103 +0000 UTC m=+3705.454813855"
Oct 29 23:45:02.189159 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.189064    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:02.940123 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.940087    5428 generic.go:356] "Generic (PLEG): container finished" podID="d8fcfb61-2011-477e-9138-c7eb186665a6" containerID="3ddbf4d1f397e2abc7c685c54643bf94ad9f0f30c028dc6f3563bf8f3482345a" exitCode=60
Oct 29 23:45:02.940570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.940175    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm" event={"ID":"d8fcfb61-2011-477e-9138-c7eb186665a6","Type":"ContainerDied","Data":"3ddbf4d1f397e2abc7c685c54643bf94ad9f0f30c028dc6f3563bf8f3482345a"}
Oct 29 23:45:02.940570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.940471    5428 scope.go:122] "RemoveContainer" containerID="3ddbf4d1f397e2abc7c685c54643bf94ad9f0f30c028dc6f3563bf8f3482345a"
Oct 29 23:45:02.942378 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.942352    5428 generic.go:356] "Generic (PLEG): container finished" podID="3ba2006e-ff86-47a3-8d81-c935530476e8" containerID="2cca1c7fcb29597c58baf82ed81d7e30c94d3bedacc61960c65e4fcbd71d0ad1" exitCode=60
Oct 29 23:45:02.942560 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.942437    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n" event={"ID":"3ba2006e-ff86-47a3-8d81-c935530476e8","Type":"ContainerDied","Data":"2cca1c7fcb29597c58baf82ed81d7e30c94d3bedacc61960c65e4fcbd71d0ad1"}
Oct 29 23:45:02.942717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.942696    5428 scope.go:122] "RemoveContainer" containerID="2cca1c7fcb29597c58baf82ed81d7e30c94d3bedacc61960c65e4fcbd71d0ad1"
Oct 29 23:45:02.946226 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.946203    5428 generic.go:356] "Generic (PLEG): container finished" podID="4ff49c0b-5f9d-411a-a767-77897583a7a3" containerID="5dd9390b739cc1870b92fc7c7efa4c4f32164b8b63f34d05b18b2de88d17f7b8" exitCode=60
Oct 29 23:45:02.946856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.946330    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4" event={"ID":"4ff49c0b-5f9d-411a-a767-77897583a7a3","Type":"ContainerDied","Data":"5dd9390b739cc1870b92fc7c7efa4c4f32164b8b63f34d05b18b2de88d17f7b8"}
Oct 29 23:45:02.946856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.946520    5428 scope.go:122] "RemoveContainer" containerID="5dd9390b739cc1870b92fc7c7efa4c4f32164b8b63f34d05b18b2de88d17f7b8"
Oct 29 23:45:02.948788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.948770    5428 generic.go:356] "Generic (PLEG): container finished" podID="36d961cc-02de-4c2d-8f84-87e66cf01259" containerID="b9db57f6877b650ffbb9b63a72fded4500bc45a081fc0f886d80452e4c7fa520" exitCode=60
Oct 29 23:45:02.948946 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.948842    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn" event={"ID":"36d961cc-02de-4c2d-8f84-87e66cf01259","Type":"ContainerDied","Data":"b9db57f6877b650ffbb9b63a72fded4500bc45a081fc0f886d80452e4c7fa520"}
Oct 29 23:45:02.949553 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.949531    5428 scope.go:122] "RemoveContainer" containerID="b9db57f6877b650ffbb9b63a72fded4500bc45a081fc0f886d80452e4c7fa520"
Oct 29 23:45:02.950846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.950825    5428 generic.go:356] "Generic (PLEG): container finished" podID="fc6e9365-f21c-49b3-b513-c17c1113cb41" containerID="ab375849628171c6666d0b7a52631a38bce4131ec1cc95cbf006945d505e3fab" exitCode=60
Oct 29 23:45:02.950971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.950848    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk" event={"ID":"fc6e9365-f21c-49b3-b513-c17c1113cb41","Type":"ContainerDied","Data":"ab375849628171c6666d0b7a52631a38bce4131ec1cc95cbf006945d505e3fab"}
Oct 29 23:45:02.951120 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.951104    5428 scope.go:122] "RemoveContainer" containerID="ab375849628171c6666d0b7a52631a38bce4131ec1cc95cbf006945d505e3fab"
Oct 29 23:45:02.952746 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.952722    5428 generic.go:356] "Generic (PLEG): container finished" podID="9ffbf666-ed06-4d62-bf9a-d1786bcf071f" containerID="aa9cbb62ac2b07c87f64b26cec81f3ed97ffdc096c890d3c7720198680eaefa0" exitCode=60
Oct 29 23:45:02.952877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.952853    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc" event={"ID":"9ffbf666-ed06-4d62-bf9a-d1786bcf071f","Type":"ContainerDied","Data":"aa9cbb62ac2b07c87f64b26cec81f3ed97ffdc096c890d3c7720198680eaefa0"}
Oct 29 23:45:02.953054 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.953037    5428 scope.go:122] "RemoveContainer" containerID="aa9cbb62ac2b07c87f64b26cec81f3ed97ffdc096c890d3c7720198680eaefa0"
Oct 29 23:45:02.957596 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.957570    5428 generic.go:356] "Generic (PLEG): container finished" podID="b0a1fa0e-f264-46bb-83c0-efd1be480dd3" containerID="fc5a606e32e1b863e050c6f065f43658609728f5625e9a3249dbbdc52b891c7a" exitCode=60
Oct 29 23:45:02.958023 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.957905    5428 scope.go:122] "RemoveContainer" containerID="fc5a606e32e1b863e050c6f065f43658609728f5625e9a3249dbbdc52b891c7a"
Oct 29 23:45:02.958177 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.957924    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727" event={"ID":"b0a1fa0e-f264-46bb-83c0-efd1be480dd3","Type":"ContainerDied","Data":"fc5a606e32e1b863e050c6f065f43658609728f5625e9a3249dbbdc52b891c7a"}
Oct 29 23:45:02.960494 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.960470    5428 generic.go:356] "Generic (PLEG): container finished" podID="918c3198-f2b2-413c-9002-91b324b0aeae" containerID="d3393123d6e5be5139fc3c2d68a6e4a4cee4d9cbf3a742a1dc067010d6973c62" exitCode=60
Oct 29 23:45:02.960590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.960569    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75" event={"ID":"918c3198-f2b2-413c-9002-91b324b0aeae","Type":"ContainerDied","Data":"d3393123d6e5be5139fc3c2d68a6e4a4cee4d9cbf3a742a1dc067010d6973c62"}
Oct 29 23:45:02.960913 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:02.960894    5428 scope.go:122] "RemoveContainer" containerID="d3393123d6e5be5139fc3c2d68a6e4a4cee4d9cbf3a742a1dc067010d6973c62"
Oct 29 23:45:03.978285 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:03.978180    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc" event={"ID":"9ffbf666-ed06-4d62-bf9a-d1786bcf071f","Type":"ContainerStarted","Data":"cb586490524cf36f343eec2bfce43ea2f12d85866d8fc8bb0de624ef85ac7ea3"}
Oct 29 23:45:03.980672 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:03.980640    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727" event={"ID":"b0a1fa0e-f264-46bb-83c0-efd1be480dd3","Type":"ContainerStarted","Data":"abd2f6e08030957a2807cd7a466e32596206524a65f5de79fe42c8ebb3759213"}
Oct 29 23:45:03.982964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:03.982939    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75" event={"ID":"918c3198-f2b2-413c-9002-91b324b0aeae","Type":"ContainerStarted","Data":"3a16cb3e916946d8ff10a637c917e1af061e8d1d02dc3d3752fd627d479c8841"}
Oct 29 23:45:03.985090 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:03.985067    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm" event={"ID":"d8fcfb61-2011-477e-9138-c7eb186665a6","Type":"ContainerStarted","Data":"bb2e2edfae507559c6c284a80190f38abe6d3f92196f4485df148256924a30a8"}
Oct 29 23:45:03.987102 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:03.987076    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n" event={"ID":"3ba2006e-ff86-47a3-8d81-c935530476e8","Type":"ContainerStarted","Data":"32ff871c8f8ea8db743ff5e4ac90d5c9fcc05ab625310bf8f63592bab7311498"}
Oct 29 23:45:03.989130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:03.989101    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4" event={"ID":"4ff49c0b-5f9d-411a-a767-77897583a7a3","Type":"ContainerStarted","Data":"0416dcb63b93e9322d927b1a5c940aad8d87c6bce1fb8ad216c2edae9f82b25a"}
Oct 29 23:45:03.991567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:03.991544    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn" event={"ID":"36d961cc-02de-4c2d-8f84-87e66cf01259","Type":"ContainerStarted","Data":"900e7131fd9d3f41bb2326238f535135284e68867a4352b2ec852cac5e915605"}
Oct 29 23:45:03.993627 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:03.993604    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk" event={"ID":"fc6e9365-f21c-49b3-b513-c17c1113cb41","Type":"ContainerStarted","Data":"03920b09fd9f59ecfd50cd818341fd994c450f9cc4dd99076201603a1eb64e18"}
Oct 29 23:45:04.024043 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:04.023981    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc" podStartSLOduration=4.023967556 podStartE2EDuration="4.023967556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:04.019035264 +0000 UTC m=+3707.361984009" watchObservedRunningTime="2026-10-29 23:45:04.023967556 +0000 UTC m=+3707.366916300"
Oct 29 23:45:04.235668 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:04.235561    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk" podStartSLOduration=4.235547026 podStartE2EDuration="4.235547026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:04.231978747 +0000 UTC m=+3707.574927493" watchObservedRunningTime="2026-10-29 23:45:04.235547026 +0000 UTC m=+3707.578496122"
Oct 29 23:45:04.292878 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:04.292850    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:04.324458 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:04.324177    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75" podStartSLOduration=4.324157216 podStartE2EDuration="4.324157216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:04.312949465 +0000 UTC m=+3707.655898211" watchObservedRunningTime="2026-10-29 23:45:04.324157216 +0000 UTC m=+3707.667105961"
Oct 29 23:45:05.064697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.064609    5428 generic.go:356] "Generic (PLEG): container finished" podID="3ba2006e-ff86-47a3-8d81-c935530476e8" containerID="32ff871c8f8ea8db743ff5e4ac90d5c9fcc05ab625310bf8f63592bab7311498" exitCode=60
Oct 29 23:45:05.065219 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.064696    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n" event={"ID":"3ba2006e-ff86-47a3-8d81-c935530476e8","Type":"ContainerDied","Data":"32ff871c8f8ea8db743ff5e4ac90d5c9fcc05ab625310bf8f63592bab7311498"}
Oct 29 23:45:05.065219 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.064753    5428 scope.go:122] "RemoveContainer" containerID="2cca1c7fcb29597c58baf82ed81d7e30c94d3bedacc61960c65e4fcbd71d0ad1"
Oct 29 23:45:05.065343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.065243    5428 scope.go:122] "RemoveContainer" containerID="32ff871c8f8ea8db743ff5e4ac90d5c9fcc05ab625310bf8f63592bab7311498"
Oct 29 23:45:05.065573 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:05.065548    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n_odh-ai-gateway-infra(3ba2006e-ff86-47a3-8d81-c935530476e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n" podUID="3ba2006e-ff86-47a3-8d81-c935530476e8"
Oct 29 23:45:05.078894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.078849    5428 generic.go:356] "Generic (PLEG): container finished" podID="4ff49c0b-5f9d-411a-a767-77897583a7a3" containerID="0416dcb63b93e9322d927b1a5c940aad8d87c6bce1fb8ad216c2edae9f82b25a" exitCode=60
Oct 29 23:45:05.079039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.078994    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4" event={"ID":"4ff49c0b-5f9d-411a-a767-77897583a7a3","Type":"ContainerDied","Data":"0416dcb63b93e9322d927b1a5c940aad8d87c6bce1fb8ad216c2edae9f82b25a"}
Oct 29 23:45:05.079466 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.079443    5428 scope.go:122] "RemoveContainer" containerID="0416dcb63b93e9322d927b1a5c940aad8d87c6bce1fb8ad216c2edae9f82b25a"
Oct 29 23:45:05.079995 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:05.079966    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4_odh-ai-gateway-infra(4ff49c0b-5f9d-411a-a767-77897583a7a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4" podUID="4ff49c0b-5f9d-411a-a767-77897583a7a3"
Oct 29 23:45:05.083706 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.083681    5428 generic.go:356] "Generic (PLEG): container finished" podID="36d961cc-02de-4c2d-8f84-87e66cf01259" containerID="900e7131fd9d3f41bb2326238f535135284e68867a4352b2ec852cac5e915605" exitCode=60
Oct 29 23:45:05.083844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.083761    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn" event={"ID":"36d961cc-02de-4c2d-8f84-87e66cf01259","Type":"ContainerDied","Data":"900e7131fd9d3f41bb2326238f535135284e68867a4352b2ec852cac5e915605"}
Oct 29 23:45:05.084208 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.084188    5428 scope.go:122] "RemoveContainer" containerID="900e7131fd9d3f41bb2326238f535135284e68867a4352b2ec852cac5e915605"
Oct 29 23:45:05.089376 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:05.084595    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn_odh-ai-gateway-infra(36d961cc-02de-4c2d-8f84-87e66cf01259)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn" podUID="36d961cc-02de-4c2d-8f84-87e66cf01259"
Oct 29 23:45:05.089376 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.087030    5428 generic.go:356] "Generic (PLEG): container finished" podID="fc6e9365-f21c-49b3-b513-c17c1113cb41" containerID="03920b09fd9f59ecfd50cd818341fd994c450f9cc4dd99076201603a1eb64e18" exitCode=60
Oct 29 23:45:05.089376 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.087139    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk" event={"ID":"fc6e9365-f21c-49b3-b513-c17c1113cb41","Type":"ContainerDied","Data":"03920b09fd9f59ecfd50cd818341fd994c450f9cc4dd99076201603a1eb64e18"}
Oct 29 23:45:05.089376 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.087498    5428 scope.go:122] "RemoveContainer" containerID="03920b09fd9f59ecfd50cd818341fd994c450f9cc4dd99076201603a1eb64e18"
Oct 29 23:45:05.089376 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:05.087839    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk_odh-ai-gateway-infra(fc6e9365-f21c-49b3-b513-c17c1113cb41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk" podUID="fc6e9365-f21c-49b3-b513-c17c1113cb41"
Oct 29 23:45:05.091138 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.090216    5428 generic.go:356] "Generic (PLEG): container finished" podID="9ffbf666-ed06-4d62-bf9a-d1786bcf071f" containerID="cb586490524cf36f343eec2bfce43ea2f12d85866d8fc8bb0de624ef85ac7ea3" exitCode=60
Oct 29 23:45:05.091138 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.090335    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc" event={"ID":"9ffbf666-ed06-4d62-bf9a-d1786bcf071f","Type":"ContainerDied","Data":"cb586490524cf36f343eec2bfce43ea2f12d85866d8fc8bb0de624ef85ac7ea3"}
Oct 29 23:45:05.091138 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.090784    5428 scope.go:122] "RemoveContainer" containerID="cb586490524cf36f343eec2bfce43ea2f12d85866d8fc8bb0de624ef85ac7ea3"
Oct 29 23:45:05.091353 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:05.091164    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc_odh-ai-gateway-infra(9ffbf666-ed06-4d62-bf9a-d1786bcf071f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc" podUID="9ffbf666-ed06-4d62-bf9a-d1786bcf071f"
Oct 29 23:45:05.095964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.094280    5428 generic.go:356] "Generic (PLEG): container finished" podID="b0a1fa0e-f264-46bb-83c0-efd1be480dd3" containerID="abd2f6e08030957a2807cd7a466e32596206524a65f5de79fe42c8ebb3759213" exitCode=60
Oct 29 23:45:05.095964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.094346    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727" event={"ID":"b0a1fa0e-f264-46bb-83c0-efd1be480dd3","Type":"ContainerDied","Data":"abd2f6e08030957a2807cd7a466e32596206524a65f5de79fe42c8ebb3759213"}
Oct 29 23:45:05.095964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.094731    5428 scope.go:122] "RemoveContainer" containerID="abd2f6e08030957a2807cd7a466e32596206524a65f5de79fe42c8ebb3759213"
Oct 29 23:45:05.095964 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:05.095060    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727_odh-ai-gateway-infra(b0a1fa0e-f264-46bb-83c0-efd1be480dd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727" podUID="b0a1fa0e-f264-46bb-83c0-efd1be480dd3"
Oct 29 23:45:05.162863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.162249    5428 generic.go:356] "Generic (PLEG): container finished" podID="918c3198-f2b2-413c-9002-91b324b0aeae" containerID="3a16cb3e916946d8ff10a637c917e1af061e8d1d02dc3d3752fd627d479c8841" exitCode=60
Oct 29 23:45:05.162863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.162495    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75" event={"ID":"918c3198-f2b2-413c-9002-91b324b0aeae","Type":"ContainerDied","Data":"3a16cb3e916946d8ff10a637c917e1af061e8d1d02dc3d3752fd627d479c8841"}
Oct 29 23:45:05.165205 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.163373    5428 scope.go:122] "RemoveContainer" containerID="3a16cb3e916946d8ff10a637c917e1af061e8d1d02dc3d3752fd627d479c8841"
Oct 29 23:45:05.165205 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:05.163880    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75_odh-ai-gateway-infra(918c3198-f2b2-413c-9002-91b324b0aeae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75" podUID="918c3198-f2b2-413c-9002-91b324b0aeae"
Oct 29 23:45:05.165767 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.165622    5428 scope.go:122] "RemoveContainer" containerID="5dd9390b739cc1870b92fc7c7efa4c4f32164b8b63f34d05b18b2de88d17f7b8"
Oct 29 23:45:05.174490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.174467    5428 generic.go:356] "Generic (PLEG): container finished" podID="d8fcfb61-2011-477e-9138-c7eb186665a6" containerID="bb2e2edfae507559c6c284a80190f38abe6d3f92196f4485df148256924a30a8" exitCode=60
Oct 29 23:45:05.174826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.174790    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm" event={"ID":"d8fcfb61-2011-477e-9138-c7eb186665a6","Type":"ContainerDied","Data":"bb2e2edfae507559c6c284a80190f38abe6d3f92196f4485df148256924a30a8"}
Oct 29 23:45:05.175340 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.175321    5428 scope.go:122] "RemoveContainer" containerID="bb2e2edfae507559c6c284a80190f38abe6d3f92196f4485df148256924a30a8"
Oct 29 23:45:05.175788 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:05.175764    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm_odh-ai-gateway-infra(d8fcfb61-2011-477e-9138-c7eb186665a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm" podUID="d8fcfb61-2011-477e-9138-c7eb186665a6"
Oct 29 23:45:05.202959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.202936    5428 scope.go:122] "RemoveContainer" containerID="b9db57f6877b650ffbb9b63a72fded4500bc45a081fc0f886d80452e4c7fa520"
Oct 29 23:45:05.226025 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.225998    5428 scope.go:122] "RemoveContainer" containerID="ab375849628171c6666d0b7a52631a38bce4131ec1cc95cbf006945d505e3fab"
Oct 29 23:45:05.245122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.245079    5428 scope.go:122] "RemoveContainer" containerID="aa9cbb62ac2b07c87f64b26cec81f3ed97ffdc096c890d3c7720198680eaefa0"
Oct 29 23:45:05.267682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.267657    5428 scope.go:122] "RemoveContainer" containerID="fc5a606e32e1b863e050c6f065f43658609728f5625e9a3249dbbdc52b891c7a"
Oct 29 23:45:05.287492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.287382    5428 scope.go:122] "RemoveContainer" containerID="d3393123d6e5be5139fc3c2d68a6e4a4cee4d9cbf3a742a1dc067010d6973c62"
Oct 29 23:45:05.306515 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:05.306478    5428 scope.go:122] "RemoveContainer" containerID="3ddbf4d1f397e2abc7c685c54643bf94ad9f0f30c028dc6f3563bf8f3482345a"
Oct 29 23:45:06.224926 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:06.224888    5428 scope.go:122] "RemoveContainer" containerID="32ff871c8f8ea8db743ff5e4ac90d5c9fcc05ab625310bf8f63592bab7311498"
Oct 29 23:45:06.225373 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:06.225336    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n_odh-ai-gateway-infra(3ba2006e-ff86-47a3-8d81-c935530476e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n" podUID="3ba2006e-ff86-47a3-8d81-c935530476e8"
Oct 29 23:45:06.228015 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:06.227990    5428 scope.go:122] "RemoveContainer" containerID="0416dcb63b93e9322d927b1a5c940aad8d87c6bce1fb8ad216c2edae9f82b25a"
Oct 29 23:45:06.228352 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:06.228329    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4_odh-ai-gateway-infra(4ff49c0b-5f9d-411a-a767-77897583a7a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4" podUID="4ff49c0b-5f9d-411a-a767-77897583a7a3"
Oct 29 23:45:06.230424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:06.230404    5428 scope.go:122] "RemoveContainer" containerID="900e7131fd9d3f41bb2326238f535135284e68867a4352b2ec852cac5e915605"
Oct 29 23:45:06.230673 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:06.230651    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn_odh-ai-gateway-infra(36d961cc-02de-4c2d-8f84-87e66cf01259)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn" podUID="36d961cc-02de-4c2d-8f84-87e66cf01259"
Oct 29 23:45:06.232501 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:06.232480    5428 scope.go:122] "RemoveContainer" containerID="03920b09fd9f59ecfd50cd818341fd994c450f9cc4dd99076201603a1eb64e18"
Oct 29 23:45:06.232732 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:06.232709    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk_odh-ai-gateway-infra(fc6e9365-f21c-49b3-b513-c17c1113cb41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk" podUID="fc6e9365-f21c-49b3-b513-c17c1113cb41"
Oct 29 23:45:06.234675 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:06.234654    5428 scope.go:122] "RemoveContainer" containerID="cb586490524cf36f343eec2bfce43ea2f12d85866d8fc8bb0de624ef85ac7ea3"
Oct 29 23:45:06.234928 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:06.234907    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc_odh-ai-gateway-infra(9ffbf666-ed06-4d62-bf9a-d1786bcf071f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc" podUID="9ffbf666-ed06-4d62-bf9a-d1786bcf071f"
Oct 29 23:45:06.236911 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:06.236890    5428 scope.go:122] "RemoveContainer" containerID="abd2f6e08030957a2807cd7a466e32596206524a65f5de79fe42c8ebb3759213"
Oct 29 23:45:06.237156 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:06.237129    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727_odh-ai-gateway-infra(b0a1fa0e-f264-46bb-83c0-efd1be480dd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727" podUID="b0a1fa0e-f264-46bb-83c0-efd1be480dd3"
Oct 29 23:45:06.241950 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:06.241915    5428 scope.go:122] "RemoveContainer" containerID="3a16cb3e916946d8ff10a637c917e1af061e8d1d02dc3d3752fd627d479c8841"
Oct 29 23:45:06.242673 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:06.242635    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75_odh-ai-gateway-infra(918c3198-f2b2-413c-9002-91b324b0aeae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75" podUID="918c3198-f2b2-413c-9002-91b324b0aeae"
Oct 29 23:45:06.244953 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:06.244937    5428 scope.go:122] "RemoveContainer" containerID="bb2e2edfae507559c6c284a80190f38abe6d3f92196f4485df148256924a30a8"
Oct 29 23:45:06.245450 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:06.245427    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm_odh-ai-gateway-infra(d8fcfb61-2011-477e-9138-c7eb186665a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm" podUID="d8fcfb61-2011-477e-9138-c7eb186665a6"
Oct 29 23:45:06.315828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:06.298032    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:07.203824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:07.203764    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:08.019671 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:08.019624    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj" podUID="9f1310de-26bc-46b9-86d6-10c8d1d49a93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:45:08.020096 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:08.019707    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd" podUID="9553cf74-cf03-4ad2-8241-d82087a823f9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:45:08.122042 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:08.121361    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m" podUID="984a1e35-8988-450a-80ad-76b3bfe60f38" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:45:09.133194 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:09.133159    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br" podUID="7850180b-94dc-4e7b-9323-b2ba77d1884d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:45:09.218823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:09.209851    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:11.095489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.095445    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:11.105097 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.105062    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"]
Oct 29 23:45:11.111968 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.111941    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.136835 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.136791    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"]
Oct 29 23:45:11.138727 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.138703    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3b8149\"/\"test-model-3b8149-kserve-self-signed-certs\""
Oct 29 23:45:11.139251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.139110    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3b8149\"/\"kube-root-ca.crt\""
Oct 29 23:45:11.148547 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.148521    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3b8149\"/\"openshift-service-ca.crt\""
Oct 29 23:45:11.148670 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.148625    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3b8149\"/\"default-dockercfg-tlzbf\""
Oct 29 23:45:11.216322 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.216284    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:11.242592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.242557    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-tmp-dir\") pod \"test-model-3b8149-kserve-746d78c69c-cp5tn\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") " pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.242770 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.242690    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7852\" (UniqueName: \"kubernetes.io/projected/f232f882-a145-4350-9c1b-94340a32b926-kube-api-access-c7852\") pod \"test-model-3b8149-kserve-746d78c69c-cp5tn\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") " pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.242770 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.242749    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-model-cache\") pod \"test-model-3b8149-kserve-746d78c69c-cp5tn\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") " pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.242944 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.242894    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-dshm\") pod \"test-model-3b8149-kserve-746d78c69c-cp5tn\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") " pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.243007 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.242942    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f232f882-a145-4350-9c1b-94340a32b926-tls-certs\") pod \"test-model-3b8149-kserve-746d78c69c-cp5tn\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") " pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.243072 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.243028    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-home\") pod \"test-model-3b8149-kserve-746d78c69c-cp5tn\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") " pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.350420 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.345963    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-dshm\") pod \"test-model-3b8149-kserve-746d78c69c-cp5tn\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") " pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.350420 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.346047    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f232f882-a145-4350-9c1b-94340a32b926-tls-certs\") pod \"test-model-3b8149-kserve-746d78c69c-cp5tn\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") " pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.350420 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.346137    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-home\") pod \"test-model-3b8149-kserve-746d78c69c-cp5tn\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") " pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.350420 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.346364    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-tmp-dir\") pod \"test-model-3b8149-kserve-746d78c69c-cp5tn\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") " pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.350420 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.346466    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7852\" (UniqueName: \"kubernetes.io/projected/f232f882-a145-4350-9c1b-94340a32b926-kube-api-access-c7852\") pod \"test-model-3b8149-kserve-746d78c69c-cp5tn\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") " pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.350420 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.346517    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-model-cache\") pod \"test-model-3b8149-kserve-746d78c69c-cp5tn\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") " pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.350420 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.347019    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-model-cache\") pod \"test-model-3b8149-kserve-746d78c69c-cp5tn\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") " pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.350420 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.349054    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-home\") pod \"test-model-3b8149-kserve-746d78c69c-cp5tn\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") " pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.350420 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.349656    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-tmp-dir\") pod \"test-model-3b8149-kserve-746d78c69c-cp5tn\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") " pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.442471 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.442381    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f232f882-a145-4350-9c1b-94340a32b926-tls-certs\") pod \"test-model-3b8149-kserve-746d78c69c-cp5tn\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") " pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.512050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.448551    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7852\" (UniqueName: \"kubernetes.io/projected/f232f882-a145-4350-9c1b-94340a32b926-kube-api-access-c7852\") pod \"test-model-3b8149-kserve-746d78c69c-cp5tn\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") " pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.512050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.452381    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-dshm\") pod \"test-model-3b8149-kserve-746d78c69c-cp5tn\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") " pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:11.742494 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:11.742457    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:12.101870 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:45:12.101838    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf232f882_a145_4350_9c1b_94340a32b926.slice/crio-576b6cab180709779c35ec643eaf8a83cc1a98d10351956f2dde0ef0ddbf8431 WatchSource:0}: Error finding container 576b6cab180709779c35ec643eaf8a83cc1a98d10351956f2dde0ef0ddbf8431: Status 404 returned error can't find the container with id 576b6cab180709779c35ec643eaf8a83cc1a98d10351956f2dde0ef0ddbf8431
Oct 29 23:45:12.103408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:12.103374    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"]
Oct 29 23:45:12.439385 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:12.439338    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn" event={"ID":"f232f882-a145-4350-9c1b-94340a32b926","Type":"ContainerStarted","Data":"97e85a160ee7cb5ed3e7cf6fc3af79d7370c8b49c91c758752afc51a34976d2e"}
Oct 29 23:45:12.439583 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:12.439392    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn" event={"ID":"f232f882-a145-4350-9c1b-94340a32b926","Type":"ContainerStarted","Data":"576b6cab180709779c35ec643eaf8a83cc1a98d10351956f2dde0ef0ddbf8431"}
Oct 29 23:45:12.483961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:12.483900    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn" podStartSLOduration=1.483883728 podStartE2EDuration="1.483883728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:12.479835225 +0000 UTC m=+3715.822783962" watchObservedRunningTime="2026-10-29 23:45:12.483883728 +0000 UTC m=+3715.826832473"
Oct 29 23:45:12.743395 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:12.743297    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:13.070202 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:13.070114    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:14.790491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:14.790461    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:15.769355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:15.769317    5428 scope.go:122] "RemoveContainer" containerID="900e7131fd9d3f41bb2326238f535135284e68867a4352b2ec852cac5e915605"
Oct 29 23:45:15.769560 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:15.769521    5428 scope.go:122] "RemoveContainer" containerID="bb2e2edfae507559c6c284a80190f38abe6d3f92196f4485df148256924a30a8"
Oct 29 23:45:15.770633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:15.770180    5428 scope.go:122] "RemoveContainer" containerID="abd2f6e08030957a2807cd7a466e32596206524a65f5de79fe42c8ebb3759213"
Oct 29 23:45:16.178536 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:16.178485    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"]
Oct 29 23:45:16.202682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:16.202623    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh" podUID="b288d408-943e-4b45-abc9-7d942e5eb3f2" containerName="maas-api" containerID="cri-o://8887ba9811f43b93a7be4876baf29c15697fde3b2c02dfc6057d3fe3f1b7725e" gracePeriod=30
Oct 29 23:45:16.253919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:16.253855    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:16.447068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:16.435274    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh" podUID="b288d408-943e-4b45-abc9-7d942e5eb3f2" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.188:8443/health\": dial tcp 10.133.0.188:8443: connect: connection refused"
Oct 29 23:45:16.798899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:16.798868    5428 scope.go:122] "RemoveContainer" containerID="03920b09fd9f59ecfd50cd818341fd994c450f9cc4dd99076201603a1eb64e18"
Oct 29 23:45:16.799175 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:16.799118    5428 scope.go:122] "RemoveContainer" containerID="cb586490524cf36f343eec2bfce43ea2f12d85866d8fc8bb0de624ef85ac7ea3"
Oct 29 23:45:17.165908 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:17.165873    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:17.285880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:17.285844    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:17.431646 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:17.429284    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:17.501724 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:17.501687    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn" event={"ID":"36d961cc-02de-4c2d-8f84-87e66cf01259","Type":"ContainerStarted","Data":"9d3038ff5ae9aaa86d28612a620f617738842e10c7fe6f7bc7035bb682d3f976"}
Oct 29 23:45:17.503495 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:17.503468    5428 generic.go:356] "Generic (PLEG): container finished" podID="b288d408-943e-4b45-abc9-7d942e5eb3f2" containerID="8887ba9811f43b93a7be4876baf29c15697fde3b2c02dfc6057d3fe3f1b7725e" exitCode=0
Oct 29 23:45:17.503667 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:17.503541    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh" event={"ID":"b288d408-943e-4b45-abc9-7d942e5eb3f2","Type":"ContainerDied","Data":"8887ba9811f43b93a7be4876baf29c15697fde3b2c02dfc6057d3fe3f1b7725e"}
Oct 29 23:45:17.506020 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:17.505996    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727" event={"ID":"b0a1fa0e-f264-46bb-83c0-efd1be480dd3","Type":"ContainerStarted","Data":"4fdfc151e47769641faa88a8462df2e51947fbac869a2637c067c412c02a4ad2"}
Oct 29 23:45:17.511752 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:17.511727    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm" event={"ID":"d8fcfb61-2011-477e-9138-c7eb186665a6","Type":"ContainerStarted","Data":"1126e81a1e5ef8ec43a66ca042f4e4e305bd76ecfb1ce657c80e50fbc2cba31d"}
Oct 29 23:45:17.770224 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:17.770169    5428 scope.go:122] "RemoveContainer" containerID="0416dcb63b93e9322d927b1a5c940aad8d87c6bce1fb8ad216c2edae9f82b25a"
Oct 29 23:45:18.011551 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.011473    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:18.017860 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:45:18.017822    5428 logging.go:55] [core] [Channel #23267 SubChannel #23269]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.181:50052", ServerName: "10.133.0.181:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.181:50052: connect: connection refused"
Oct 29 23:45:18.017860 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:45:18.017852    5428 logging.go:55] [core] [Channel #23266 SubChannel #23268]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.182:50052", ServerName: "10.133.0.182:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.182:50052: connect: connection refused"
Oct 29 23:45:18.085441 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:18.085400    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:45:18.111745 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.111714    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m" podUID="984a1e35-8988-450a-80ad-76b3bfe60f38" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:45:18.503521 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.503487    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:45:18.534992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.534912    5428 generic.go:356] "Generic (PLEG): container finished" podID="9f1310de-26bc-46b9-86d6-10c8d1d49a93" containerID="cbe4b6348319836c795de7c18c0186ebf073f26d8ab851f7fb7a09cb8a776c82" exitCode=0
Oct 29 23:45:18.535150 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.535038    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj" event={"ID":"9f1310de-26bc-46b9-86d6-10c8d1d49a93","Type":"ContainerDied","Data":"cbe4b6348319836c795de7c18c0186ebf073f26d8ab851f7fb7a09cb8a776c82"}
Oct 29 23:45:18.537121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.537089    5428 generic.go:356] "Generic (PLEG): container finished" podID="9553cf74-cf03-4ad2-8241-d82087a823f9" containerID="9473cc06c02a0b13020b16e281c4f2215afbf64e24a22ab3e9c07263362db820" exitCode=0
Oct 29 23:45:18.537265 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.537128    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd" event={"ID":"9553cf74-cf03-4ad2-8241-d82087a823f9","Type":"ContainerDied","Data":"9473cc06c02a0b13020b16e281c4f2215afbf64e24a22ab3e9c07263362db820"}
Oct 29 23:45:18.540562 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.540530    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk" event={"ID":"fc6e9365-f21c-49b3-b513-c17c1113cb41","Type":"ContainerStarted","Data":"463fac0662eed54a92b6535d84884c63237e705ef594ab050fdd2978c427ff4c"}
Oct 29 23:45:18.542288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.542260    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh" event={"ID":"b288d408-943e-4b45-abc9-7d942e5eb3f2","Type":"ContainerDied","Data":"64e999f02ab4407c7e6b3fb2fbe722b3690d3d9aa19ff3aaa557e23f1c4cb2ba"}
Oct 29 23:45:18.542378 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.542319    5428 scope.go:122] "RemoveContainer" containerID="8887ba9811f43b93a7be4876baf29c15697fde3b2c02dfc6057d3fe3f1b7725e"
Oct 29 23:45:18.542496 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.542478    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"
Oct 29 23:45:18.546652 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.546585    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"
Oct 29 23:45:18.562871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.546205    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc" event={"ID":"9ffbf666-ed06-4d62-bf9a-d1786bcf071f","Type":"ContainerStarted","Data":"036789793a2decf04b33658808d7b71acf87854ef96192bfcd29cbe32b7a917b"}
Oct 29 23:45:18.575864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.575833    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn"]
Oct 29 23:45:18.576075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.576049    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn" podUID="36d961cc-02de-4c2d-8f84-87e66cf01259" containerName="cleanup" containerID="cri-o://9d3038ff5ae9aaa86d28612a620f617738842e10c7fe6f7bc7035bb682d3f976" gracePeriod=30
Oct 29 23:45:18.622233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.622203    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727"]
Oct 29 23:45:18.622447 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.622426    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727" podUID="b0a1fa0e-f264-46bb-83c0-efd1be480dd3" containerName="cleanup" containerID="cri-o://4fdfc151e47769641faa88a8462df2e51947fbac869a2637c067c412c02a4ad2" gracePeriod=30
Oct 29 23:45:18.633786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.633755    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b288d408-943e-4b45-abc9-7d942e5eb3f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b288d408-943e-4b45-abc9-7d942e5eb3f2-openshift-service-ca\") pod \"b288d408-943e-4b45-abc9-7d942e5eb3f2\" (UID: \"b288d408-943e-4b45-abc9-7d942e5eb3f2\") "
Oct 29 23:45:18.633991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.633972    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b288d408-943e-4b45-abc9-7d942e5eb3f2-kube-api-access-sptlh\" (UniqueName: \"kubernetes.io/projected/b288d408-943e-4b45-abc9-7d942e5eb3f2-kube-api-access-sptlh\") pod \"b288d408-943e-4b45-abc9-7d942e5eb3f2\" (UID: \"b288d408-943e-4b45-abc9-7d942e5eb3f2\") "
Oct 29 23:45:18.634077 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.634023    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b288d408-943e-4b45-abc9-7d942e5eb3f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b288d408-943e-4b45-abc9-7d942e5eb3f2-metrics-certs\") pod \"b288d408-943e-4b45-abc9-7d942e5eb3f2\" (UID: \"b288d408-943e-4b45-abc9-7d942e5eb3f2\") "
Oct 29 23:45:18.634170 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.634153    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b288d408-943e-4b45-abc9-7d942e5eb3f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b288d408-943e-4b45-abc9-7d942e5eb3f2-maas-api-tls\") pod \"b288d408-943e-4b45-abc9-7d942e5eb3f2\" (UID: \"b288d408-943e-4b45-abc9-7d942e5eb3f2\") "
Oct 29 23:45:18.665028 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.659671    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b288d408-943e-4b45-abc9-7d942e5eb3f2-openshift-service-ca" pod "b288d408-943e-4b45-abc9-7d942e5eb3f2" (UID: "b288d408-943e-4b45-abc9-7d942e5eb3f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:18.672335 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.671178    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b288d408-943e-4b45-abc9-7d942e5eb3f2-kube-api-access-sptlh" pod "b288d408-943e-4b45-abc9-7d942e5eb3f2" (UID: "b288d408-943e-4b45-abc9-7d942e5eb3f2"). InnerVolumeSpecName "kube-api-access-sptlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:18.678659 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.678493    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b288d408-943e-4b45-abc9-7d942e5eb3f2-maas-api-tls" pod "b288d408-943e-4b45-abc9-7d942e5eb3f2" (UID: "b288d408-943e-4b45-abc9-7d942e5eb3f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:45:18.688952 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.688923    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b288d408-943e-4b45-abc9-7d942e5eb3f2-metrics-certs" pod "b288d408-943e-4b45-abc9-7d942e5eb3f2" (UID: "b288d408-943e-4b45-abc9-7d942e5eb3f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:45:18.700075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.700045    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm"]
Oct 29 23:45:18.700348 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.700312    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm" podUID="d8fcfb61-2011-477e-9138-c7eb186665a6" containerName="cleanup" containerID="cri-o://1126e81a1e5ef8ec43a66ca042f4e4e305bd76ecfb1ce657c80e50fbc2cba31d" gracePeriod=30
Oct 29 23:45:18.772051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.770372    5428 scope.go:122] "RemoveContainer" containerID="3a16cb3e916946d8ff10a637c917e1af061e8d1d02dc3d3752fd627d479c8841"
Oct 29 23:45:18.772487 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.772116    5428 scope.go:122] "RemoveContainer" containerID="32ff871c8f8ea8db743ff5e4ac90d5c9fcc05ab625310bf8f63592bab7311498"
Oct 29 23:45:18.786122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.775622    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f1310de-26bc-46b9-86d6-10c8d1d49a93-kube-api-access-dtkcl\" (UniqueName: \"kubernetes.io/projected/9f1310de-26bc-46b9-86d6-10c8d1d49a93-kube-api-access-dtkcl\") pod \"9f1310de-26bc-46b9-86d6-10c8d1d49a93\" (UID: \"9f1310de-26bc-46b9-86d6-10c8d1d49a93\") "
Oct 29 23:45:18.786122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.775666    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f1310de-26bc-46b9-86d6-10c8d1d49a93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f1310de-26bc-46b9-86d6-10c8d1d49a93-plugins-config-volume\") pod \"9f1310de-26bc-46b9-86d6-10c8d1d49a93\" (UID: \"9f1310de-26bc-46b9-86d6-10c8d1d49a93\") "
Oct 29 23:45:18.786122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.776336    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sptlh\" (UniqueName: \"kubernetes.io/projected/b288d408-943e-4b45-abc9-7d942e5eb3f2-kube-api-access-sptlh\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:18.786122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.776360    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b288d408-943e-4b45-abc9-7d942e5eb3f2-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:18.786122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.776375    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b288d408-943e-4b45-abc9-7d942e5eb3f2-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:18.786122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.776387    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b288d408-943e-4b45-abc9-7d942e5eb3f2-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:18.786122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.777557    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f1310de-26bc-46b9-86d6-10c8d1d49a93-plugins-config-volume" pod "9f1310de-26bc-46b9-86d6-10c8d1d49a93" (UID: "9f1310de-26bc-46b9-86d6-10c8d1d49a93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:18.788062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.787497    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f1310de-26bc-46b9-86d6-10c8d1d49a93-kube-api-access-dtkcl" pod "9f1310de-26bc-46b9-86d6-10c8d1d49a93" (UID: "9f1310de-26bc-46b9-86d6-10c8d1d49a93"). InnerVolumeSpecName "kube-api-access-dtkcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:18.788850 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.788663    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:18.923831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.909410    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtkcl\" (UniqueName: \"kubernetes.io/projected/9f1310de-26bc-46b9-86d6-10c8d1d49a93-kube-api-access-dtkcl\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:18.923831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.917937    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f1310de-26bc-46b9-86d6-10c8d1d49a93-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:18.937917 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:18.930832    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:19.025053 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.022112    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd" podUID="9553cf74-cf03-4ad2-8241-d82087a823f9" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.181:50052\" within 1s: context deadline exceeded"
Oct 29 23:45:19.025053 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.022270    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"
Oct 29 23:45:19.025053 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:45:19.022714    5428 logging.go:55] [core] [Channel #23267 SubChannel #23269]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.181:50052", ServerName: "10.133.0.181:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.181:50052: operation was canceled"
Oct 29 23:45:19.025053 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.024818    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj" podUID="9f1310de-26bc-46b9-86d6-10c8d1d49a93" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.182:50052\" within 1s: context deadline exceeded"
Oct 29 23:45:19.025053 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.024894    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"
Oct 29 23:45:19.025053 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:45:19.025010    5428 logging.go:55] [core] [Channel #23266 SubChannel #23268]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.182:50052", ServerName: "10.133.0.182:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.182:50052: operation was canceled"
Oct 29 23:45:19.097367 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.097075    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"
Oct 29 23:45:19.135681 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.135625    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9553cf74-cf03-4ad2-8241-d82087a823f9-kube-api-access-bznkp\" (UniqueName: \"kubernetes.io/projected/9553cf74-cf03-4ad2-8241-d82087a823f9-kube-api-access-bznkp\") pod \"9553cf74-cf03-4ad2-8241-d82087a823f9\" (UID: \"9553cf74-cf03-4ad2-8241-d82087a823f9\") "
Oct 29 23:45:19.137555 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.136872    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9553cf74-cf03-4ad2-8241-d82087a823f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9553cf74-cf03-4ad2-8241-d82087a823f9-plugins-config-volume\") pod \"9553cf74-cf03-4ad2-8241-d82087a823f9\" (UID: \"9553cf74-cf03-4ad2-8241-d82087a823f9\") "
Oct 29 23:45:19.140200 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.140059    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9553cf74-cf03-4ad2-8241-d82087a823f9-plugins-config-volume" pod "9553cf74-cf03-4ad2-8241-d82087a823f9" (UID: "9553cf74-cf03-4ad2-8241-d82087a823f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:19.142099 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.142059    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9553cf74-cf03-4ad2-8241-d82087a823f9-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:19.143293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.142881    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br" podUID="7850180b-94dc-4e7b-9323-b2ba77d1884d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:45:19.149214 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.149193    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"]
Oct 29 23:45:19.153577 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.153443    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9553cf74-cf03-4ad2-8241-d82087a823f9-kube-api-access-bznkp" pod "9553cf74-cf03-4ad2-8241-d82087a823f9" (UID: "9553cf74-cf03-4ad2-8241-d82087a823f9"). InnerVolumeSpecName "kube-api-access-bznkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:19.155245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.155223    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727"
Oct 29 23:45:19.157343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.157317    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b73a75-7b885ff5f9-r8dnh"]
Oct 29 23:45:19.169248 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.169222    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn"
Oct 29 23:45:19.218123 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.218090    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm"
Oct 29 23:45:19.242971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.242924    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d961cc-02de-4c2d-8f84-87e66cf01259-kube-api-access-wtcrt\" (UniqueName: \"kubernetes.io/projected/36d961cc-02de-4c2d-8f84-87e66cf01259-kube-api-access-wtcrt\") pod \"36d961cc-02de-4c2d-8f84-87e66cf01259\" (UID: \"36d961cc-02de-4c2d-8f84-87e66cf01259\") "
Oct 29 23:45:19.244039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.242995    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a1fa0e-f264-46bb-83c0-efd1be480dd3-kube-api-access-sd5qh\" (UniqueName: \"kubernetes.io/projected/b0a1fa0e-f264-46bb-83c0-efd1be480dd3-kube-api-access-sd5qh\") pod \"b0a1fa0e-f264-46bb-83c0-efd1be480dd3\" (UID: \"b0a1fa0e-f264-46bb-83c0-efd1be480dd3\") "
Oct 29 23:45:19.244039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.243854    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36d961cc-02de-4c2d-8f84-87e66cf01259-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d961cc-02de-4c2d-8f84-87e66cf01259-maas-api-service-ca\") pod \"36d961cc-02de-4c2d-8f84-87e66cf01259\" (UID: \"36d961cc-02de-4c2d-8f84-87e66cf01259\") "
Oct 29 23:45:19.244296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.244274    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0a1fa0e-f264-46bb-83c0-efd1be480dd3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a1fa0e-f264-46bb-83c0-efd1be480dd3-maas-api-service-ca\") pod \"b0a1fa0e-f264-46bb-83c0-efd1be480dd3\" (UID: \"b0a1fa0e-f264-46bb-83c0-efd1be480dd3\") "
Oct 29 23:45:19.245166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.245138    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36d961cc-02de-4c2d-8f84-87e66cf01259-maas-api-service-ca" pod "36d961cc-02de-4c2d-8f84-87e66cf01259" (UID: "36d961cc-02de-4c2d-8f84-87e66cf01259"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:19.245517 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.245492    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a1fa0e-f264-46bb-83c0-efd1be480dd3-maas-api-service-ca" pod "b0a1fa0e-f264-46bb-83c0-efd1be480dd3" (UID: "b0a1fa0e-f264-46bb-83c0-efd1be480dd3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:19.246037 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.246011    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a1fa0e-f264-46bb-83c0-efd1be480dd3-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:19.246153 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.246039    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d961cc-02de-4c2d-8f84-87e66cf01259-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:19.246153 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.246053    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bznkp\" (UniqueName: \"kubernetes.io/projected/9553cf74-cf03-4ad2-8241-d82087a823f9-kube-api-access-bznkp\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:19.250700 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.250671    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d961cc-02de-4c2d-8f84-87e66cf01259-kube-api-access-wtcrt" pod "36d961cc-02de-4c2d-8f84-87e66cf01259" (UID: "36d961cc-02de-4c2d-8f84-87e66cf01259"). InnerVolumeSpecName "kube-api-access-wtcrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:19.253372 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.253337    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a1fa0e-f264-46bb-83c0-efd1be480dd3-kube-api-access-sd5qh" pod "b0a1fa0e-f264-46bb-83c0-efd1be480dd3" (UID: "b0a1fa0e-f264-46bb-83c0-efd1be480dd3"). InnerVolumeSpecName "kube-api-access-sd5qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:19.347223 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.346717    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8fcfb61-2011-477e-9138-c7eb186665a6-kube-api-access-d2npg\" (UniqueName: \"kubernetes.io/projected/d8fcfb61-2011-477e-9138-c7eb186665a6-kube-api-access-d2npg\") pod \"d8fcfb61-2011-477e-9138-c7eb186665a6\" (UID: \"d8fcfb61-2011-477e-9138-c7eb186665a6\") "
Oct 29 23:45:19.347223 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.346947    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8fcfb61-2011-477e-9138-c7eb186665a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8fcfb61-2011-477e-9138-c7eb186665a6-maas-api-service-ca\") pod \"d8fcfb61-2011-477e-9138-c7eb186665a6\" (UID: \"d8fcfb61-2011-477e-9138-c7eb186665a6\") "
Oct 29 23:45:19.382590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.347839    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtcrt\" (UniqueName: \"kubernetes.io/projected/36d961cc-02de-4c2d-8f84-87e66cf01259-kube-api-access-wtcrt\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:19.382590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.347868    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd5qh\" (UniqueName: \"kubernetes.io/projected/b0a1fa0e-f264-46bb-83c0-efd1be480dd3-kube-api-access-sd5qh\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:19.382590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.355115    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8fcfb61-2011-477e-9138-c7eb186665a6-maas-api-service-ca" pod "d8fcfb61-2011-477e-9138-c7eb186665a6" (UID: "d8fcfb61-2011-477e-9138-c7eb186665a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:19.382590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.368208    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fcfb61-2011-477e-9138-c7eb186665a6-kube-api-access-d2npg" pod "d8fcfb61-2011-477e-9138-c7eb186665a6" (UID: "d8fcfb61-2011-477e-9138-c7eb186665a6"). InnerVolumeSpecName "kube-api-access-d2npg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:19.449386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.449351    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2npg\" (UniqueName: \"kubernetes.io/projected/d8fcfb61-2011-477e-9138-c7eb186665a6-kube-api-access-d2npg\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:19.449547 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.449390    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8fcfb61-2011-477e-9138-c7eb186665a6-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:19.558275 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.558230    5428 generic.go:356] "Generic (PLEG): container finished" podID="36d961cc-02de-4c2d-8f84-87e66cf01259" containerID="9d3038ff5ae9aaa86d28612a620f617738842e10c7fe6f7bc7035bb682d3f976" exitCode=60
Oct 29 23:45:19.558788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.558287    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn"
Oct 29 23:45:19.558788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.558315    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn" event={"ID":"36d961cc-02de-4c2d-8f84-87e66cf01259","Type":"ContainerDied","Data":"9d3038ff5ae9aaa86d28612a620f617738842e10c7fe6f7bc7035bb682d3f976"}
Oct 29 23:45:19.558788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.558386    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn" event={"ID":"36d961cc-02de-4c2d-8f84-87e66cf01259","Type":"ContainerDied","Data":"d2d04900eeab1832c55f55698b44f3ef1db9b2946b18a82e24abd761fab4b594"}
Oct 29 23:45:19.558788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.558419    5428 scope.go:122] "RemoveContainer" containerID="9d3038ff5ae9aaa86d28612a620f617738842e10c7fe6f7bc7035bb682d3f976"
Oct 29 23:45:19.561629 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.561589    5428 generic.go:356] "Generic (PLEG): container finished" podID="fc6e9365-f21c-49b3-b513-c17c1113cb41" containerID="463fac0662eed54a92b6535d84884c63237e705ef594ab050fdd2978c427ff4c" exitCode=60
Oct 29 23:45:19.561747 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.561690    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk" event={"ID":"fc6e9365-f21c-49b3-b513-c17c1113cb41","Type":"ContainerDied","Data":"463fac0662eed54a92b6535d84884c63237e705ef594ab050fdd2978c427ff4c"}
Oct 29 23:45:19.562193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.562173    5428 scope.go:122] "RemoveContainer" containerID="463fac0662eed54a92b6535d84884c63237e705ef594ab050fdd2978c427ff4c"
Oct 29 23:45:19.562787 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:19.562764    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk_odh-ai-gateway-infra(fc6e9365-f21c-49b3-b513-c17c1113cb41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk" podUID="fc6e9365-f21c-49b3-b513-c17c1113cb41"
Oct 29 23:45:19.566098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.566076    5428 generic.go:356] "Generic (PLEG): container finished" podID="9ffbf666-ed06-4d62-bf9a-d1786bcf071f" containerID="036789793a2decf04b33658808d7b71acf87854ef96192bfcd29cbe32b7a917b" exitCode=60
Oct 29 23:45:19.566181 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.566130    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc" event={"ID":"9ffbf666-ed06-4d62-bf9a-d1786bcf071f","Type":"ContainerDied","Data":"036789793a2decf04b33658808d7b71acf87854ef96192bfcd29cbe32b7a917b"}
Oct 29 23:45:19.566604 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.566561    5428 scope.go:122] "RemoveContainer" containerID="036789793a2decf04b33658808d7b71acf87854ef96192bfcd29cbe32b7a917b"
Oct 29 23:45:19.567009 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:19.566983    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc_odh-ai-gateway-infra(9ffbf666-ed06-4d62-bf9a-d1786bcf071f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc" podUID="9ffbf666-ed06-4d62-bf9a-d1786bcf071f"
Oct 29 23:45:19.568021 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.568000    5428 generic.go:356] "Generic (PLEG): container finished" podID="b0a1fa0e-f264-46bb-83c0-efd1be480dd3" containerID="4fdfc151e47769641faa88a8462df2e51947fbac869a2637c067c412c02a4ad2" exitCode=60
Oct 29 23:45:19.568116 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.568078    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727"
Oct 29 23:45:19.568116 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.568102    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727" event={"ID":"b0a1fa0e-f264-46bb-83c0-efd1be480dd3","Type":"ContainerDied","Data":"4fdfc151e47769641faa88a8462df2e51947fbac869a2637c067c412c02a4ad2"}
Oct 29 23:45:19.568228 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.568149    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727" event={"ID":"b0a1fa0e-f264-46bb-83c0-efd1be480dd3","Type":"ContainerDied","Data":"970ae0cc3b75927e1e47667cac259f996b8054fb33d1f5dd2e071c23881909bc"}
Oct 29 23:45:19.570885 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.570664    5428 generic.go:356] "Generic (PLEG): container finished" podID="d8fcfb61-2011-477e-9138-c7eb186665a6" containerID="1126e81a1e5ef8ec43a66ca042f4e4e305bd76ecfb1ce657c80e50fbc2cba31d" exitCode=60
Oct 29 23:45:19.570885 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.570748    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm"
Oct 29 23:45:19.570885 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.570782    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm" event={"ID":"d8fcfb61-2011-477e-9138-c7eb186665a6","Type":"ContainerDied","Data":"1126e81a1e5ef8ec43a66ca042f4e4e305bd76ecfb1ce657c80e50fbc2cba31d"}
Oct 29 23:45:19.570885 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.570835    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm" event={"ID":"d8fcfb61-2011-477e-9138-c7eb186665a6","Type":"ContainerDied","Data":"2f7f6fae4bd862227f48bbc277d8087b44105a85c3d73512d6301544779d8c4d"}
Oct 29 23:45:19.579479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.579457    5428 scope.go:122] "RemoveContainer" containerID="900e7131fd9d3f41bb2326238f535135284e68867a4352b2ec852cac5e915605"
Oct 29 23:45:19.581144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.581112    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj" event={"ID":"9f1310de-26bc-46b9-86d6-10c8d1d49a93","Type":"ContainerDied","Data":"f75c253bae1a18b475367f0e5677bc2c7aefa72f95415b328af1c6bacfd48efe"}
Oct 29 23:45:19.581245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.581180    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"
Oct 29 23:45:19.583135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.583109    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd" event={"ID":"9553cf74-cf03-4ad2-8241-d82087a823f9","Type":"ContainerDied","Data":"f93841df843cba48f8c28b7339d44a84bc0406ae2fb7b4c9df0b683e42e149a5"}
Oct 29 23:45:19.583246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.583199    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"
Oct 29 23:45:19.587416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.587393    5428 generic.go:356] "Generic (PLEG): container finished" podID="4ff49c0b-5f9d-411a-a767-77897583a7a3" containerID="8234ac5cc0e01ccac16e0b6a17f17c1697aabad1e103b04d98205368c7121404" exitCode=60
Oct 29 23:45:19.587535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.587444    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4" event={"ID":"4ff49c0b-5f9d-411a-a767-77897583a7a3","Type":"ContainerDied","Data":"8234ac5cc0e01ccac16e0b6a17f17c1697aabad1e103b04d98205368c7121404"}
Oct 29 23:45:19.587855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.587835    5428 scope.go:122] "RemoveContainer" containerID="8234ac5cc0e01ccac16e0b6a17f17c1697aabad1e103b04d98205368c7121404"
Oct 29 23:45:19.588078 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:19.588059    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4_odh-ai-gateway-infra(4ff49c0b-5f9d-411a-a767-77897583a7a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4" podUID="4ff49c0b-5f9d-411a-a767-77897583a7a3"
Oct 29 23:45:19.600484 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.600467    5428 scope.go:122] "RemoveContainer" containerID="9d3038ff5ae9aaa86d28612a620f617738842e10c7fe6f7bc7035bb682d3f976"
Oct 29 23:45:19.600738 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:19.600716    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3038ff5ae9aaa86d28612a620f617738842e10c7fe6f7bc7035bb682d3f976\": container with ID starting with 9d3038ff5ae9aaa86d28612a620f617738842e10c7fe6f7bc7035bb682d3f976 not found: ID does not exist" containerID="9d3038ff5ae9aaa86d28612a620f617738842e10c7fe6f7bc7035bb682d3f976"
Oct 29 23:45:19.600830 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.600749    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3038ff5ae9aaa86d28612a620f617738842e10c7fe6f7bc7035bb682d3f976"} err="failed to get container status \"9d3038ff5ae9aaa86d28612a620f617738842e10c7fe6f7bc7035bb682d3f976\": rpc error: code = NotFound desc = could not find container \"9d3038ff5ae9aaa86d28612a620f617738842e10c7fe6f7bc7035bb682d3f976\": container with ID starting with 9d3038ff5ae9aaa86d28612a620f617738842e10c7fe6f7bc7035bb682d3f976 not found: ID does not exist"
Oct 29 23:45:19.600830 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.600769    5428 scope.go:122] "RemoveContainer" containerID="900e7131fd9d3f41bb2326238f535135284e68867a4352b2ec852cac5e915605"
Oct 29 23:45:19.601067 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:19.601046    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"900e7131fd9d3f41bb2326238f535135284e68867a4352b2ec852cac5e915605\": container with ID starting with 900e7131fd9d3f41bb2326238f535135284e68867a4352b2ec852cac5e915605 not found: ID does not exist" containerID="900e7131fd9d3f41bb2326238f535135284e68867a4352b2ec852cac5e915605"
Oct 29 23:45:19.601133 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.601074    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"900e7131fd9d3f41bb2326238f535135284e68867a4352b2ec852cac5e915605"} err="failed to get container status \"900e7131fd9d3f41bb2326238f535135284e68867a4352b2ec852cac5e915605\": rpc error: code = NotFound desc = could not find container \"900e7131fd9d3f41bb2326238f535135284e68867a4352b2ec852cac5e915605\": container with ID starting with 900e7131fd9d3f41bb2326238f535135284e68867a4352b2ec852cac5e915605 not found: ID does not exist"
Oct 29 23:45:19.601133 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.601088    5428 scope.go:122] "RemoveContainer" containerID="03920b09fd9f59ecfd50cd818341fd994c450f9cc4dd99076201603a1eb64e18"
Oct 29 23:45:19.625933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.625914    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn"]
Oct 29 23:45:19.632554 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.632534    5428 scope.go:122] "RemoveContainer" containerID="cb586490524cf36f343eec2bfce43ea2f12d85866d8fc8bb0de624ef85ac7ea3"
Oct 29 23:45:19.633588 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.633535    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-adc7d7-29888625-6d2rn"]
Oct 29 23:45:19.660170 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.660143    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk"]
Oct 29 23:45:19.782137 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.782078    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"]
Oct 29 23:45:19.788996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.788970    5428 scope.go:122] "RemoveContainer" containerID="4fdfc151e47769641faa88a8462df2e51947fbac869a2637c067c412c02a4ad2"
Oct 29 23:45:19.791460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.791434    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b73a75-7f87b9dcbb-ftlvj"]
Oct 29 23:45:19.807883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.807854    5428 scope.go:122] "RemoveContainer" containerID="abd2f6e08030957a2807cd7a466e32596206524a65f5de79fe42c8ebb3759213"
Oct 29 23:45:19.840617 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.840313    5428 scope.go:122] "RemoveContainer" containerID="4fdfc151e47769641faa88a8462df2e51947fbac869a2637c067c412c02a4ad2"
Oct 29 23:45:19.861440 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:19.861351    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fdfc151e47769641faa88a8462df2e51947fbac869a2637c067c412c02a4ad2\": container with ID starting with 4fdfc151e47769641faa88a8462df2e51947fbac869a2637c067c412c02a4ad2 not found: ID does not exist" containerID="4fdfc151e47769641faa88a8462df2e51947fbac869a2637c067c412c02a4ad2"
Oct 29 23:45:19.861440 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.861400    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fdfc151e47769641faa88a8462df2e51947fbac869a2637c067c412c02a4ad2"} err="failed to get container status \"4fdfc151e47769641faa88a8462df2e51947fbac869a2637c067c412c02a4ad2\": rpc error: code = NotFound desc = could not find container \"4fdfc151e47769641faa88a8462df2e51947fbac869a2637c067c412c02a4ad2\": container with ID starting with 4fdfc151e47769641faa88a8462df2e51947fbac869a2637c067c412c02a4ad2 not found: ID does not exist"
Oct 29 23:45:19.861440 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.861427    5428 scope.go:122] "RemoveContainer" containerID="abd2f6e08030957a2807cd7a466e32596206524a65f5de79fe42c8ebb3759213"
Oct 29 23:45:19.865458 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:19.864700    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abd2f6e08030957a2807cd7a466e32596206524a65f5de79fe42c8ebb3759213\": container with ID starting with abd2f6e08030957a2807cd7a466e32596206524a65f5de79fe42c8ebb3759213 not found: ID does not exist" containerID="abd2f6e08030957a2807cd7a466e32596206524a65f5de79fe42c8ebb3759213"
Oct 29 23:45:19.865458 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.864743    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abd2f6e08030957a2807cd7a466e32596206524a65f5de79fe42c8ebb3759213"} err="failed to get container status \"abd2f6e08030957a2807cd7a466e32596206524a65f5de79fe42c8ebb3759213\": rpc error: code = NotFound desc = could not find container \"abd2f6e08030957a2807cd7a466e32596206524a65f5de79fe42c8ebb3759213\": container with ID starting with abd2f6e08030957a2807cd7a466e32596206524a65f5de79fe42c8ebb3759213 not found: ID does not exist"
Oct 29 23:45:19.865458 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.864768    5428 scope.go:122] "RemoveContainer" containerID="1126e81a1e5ef8ec43a66ca042f4e4e305bd76ecfb1ce657c80e50fbc2cba31d"
Oct 29 23:45:19.867553 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.867487    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727"]
Oct 29 23:45:19.877837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.877409    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed8218-29888625-p2727"]
Oct 29 23:45:19.880779 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.880759    5428 scope.go:122] "RemoveContainer" containerID="bb2e2edfae507559c6c284a80190f38abe6d3f92196f4485df148256924a30a8"
Oct 29 23:45:19.885551 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.885525    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc"]
Oct 29 23:45:19.915083 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.915055    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm"]
Oct 29 23:45:19.915214 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.915147    5428 scope.go:122] "RemoveContainer" containerID="1126e81a1e5ef8ec43a66ca042f4e4e305bd76ecfb1ce657c80e50fbc2cba31d"
Oct 29 23:45:19.915547 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:19.915520    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1126e81a1e5ef8ec43a66ca042f4e4e305bd76ecfb1ce657c80e50fbc2cba31d\": container with ID starting with 1126e81a1e5ef8ec43a66ca042f4e4e305bd76ecfb1ce657c80e50fbc2cba31d not found: ID does not exist" containerID="1126e81a1e5ef8ec43a66ca042f4e4e305bd76ecfb1ce657c80e50fbc2cba31d"
Oct 29 23:45:19.915602 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.915559    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1126e81a1e5ef8ec43a66ca042f4e4e305bd76ecfb1ce657c80e50fbc2cba31d"} err="failed to get container status \"1126e81a1e5ef8ec43a66ca042f4e4e305bd76ecfb1ce657c80e50fbc2cba31d\": rpc error: code = NotFound desc = could not find container \"1126e81a1e5ef8ec43a66ca042f4e4e305bd76ecfb1ce657c80e50fbc2cba31d\": container with ID starting with 1126e81a1e5ef8ec43a66ca042f4e4e305bd76ecfb1ce657c80e50fbc2cba31d not found: ID does not exist"
Oct 29 23:45:19.915602 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.915584    5428 scope.go:122] "RemoveContainer" containerID="bb2e2edfae507559c6c284a80190f38abe6d3f92196f4485df148256924a30a8"
Oct 29 23:45:19.915929 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:19.915904    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2e2edfae507559c6c284a80190f38abe6d3f92196f4485df148256924a30a8\": container with ID starting with bb2e2edfae507559c6c284a80190f38abe6d3f92196f4485df148256924a30a8 not found: ID does not exist" containerID="bb2e2edfae507559c6c284a80190f38abe6d3f92196f4485df148256924a30a8"
Oct 29 23:45:19.915967 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.915938    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2e2edfae507559c6c284a80190f38abe6d3f92196f4485df148256924a30a8"} err="failed to get container status \"bb2e2edfae507559c6c284a80190f38abe6d3f92196f4485df148256924a30a8\": rpc error: code = NotFound desc = could not find container \"bb2e2edfae507559c6c284a80190f38abe6d3f92196f4485df148256924a30a8\": container with ID starting with bb2e2edfae507559c6c284a80190f38abe6d3f92196f4485df148256924a30a8 not found: ID does not exist"
Oct 29 23:45:19.915967 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.915961    5428 scope.go:122] "RemoveContainer" containerID="cbe4b6348319836c795de7c18c0186ebf073f26d8ab851f7fb7a09cb8a776c82"
Oct 29 23:45:19.919536 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.919376    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7aeaad-29888625-ltfxm"]
Oct 29 23:45:19.948067 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.947987    5428 scope.go:122] "RemoveContainer" containerID="9473cc06c02a0b13020b16e281c4f2215afbf64e24a22ab3e9c07263362db820"
Oct 29 23:45:19.958196 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.958171    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"]
Oct 29 23:45:19.963976 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.963935    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b73a75-64bc775475-phcjd"]
Oct 29 23:45:19.980267 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.980212    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:19.999099 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:19.999079    5428 scope.go:122] "RemoveContainer" containerID="0416dcb63b93e9322d927b1a5c940aad8d87c6bce1fb8ad216c2edae9f82b25a"
Oct 29 23:45:20.027089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.027053    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br"
Oct 29 23:45:20.117723 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.117689    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m"
Oct 29 23:45:20.168592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.167692    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7850180b-94dc-4e7b-9323-b2ba77d1884d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7850180b-94dc-4e7b-9323-b2ba77d1884d-plugins-config-volume\") pod \"7850180b-94dc-4e7b-9323-b2ba77d1884d\" (UID: \"7850180b-94dc-4e7b-9323-b2ba77d1884d\") "
Oct 29 23:45:20.168592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.167775    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984a1e35-8988-450a-80ad-76b3bfe60f38-kube-api-access-p5tvx\" (UniqueName: \"kubernetes.io/projected/984a1e35-8988-450a-80ad-76b3bfe60f38-kube-api-access-p5tvx\") pod \"984a1e35-8988-450a-80ad-76b3bfe60f38\" (UID: \"984a1e35-8988-450a-80ad-76b3bfe60f38\") "
Oct 29 23:45:20.168592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.167962    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7850180b-94dc-4e7b-9323-b2ba77d1884d-kube-api-access-qftgd\" (UniqueName: \"kubernetes.io/projected/7850180b-94dc-4e7b-9323-b2ba77d1884d-kube-api-access-qftgd\") pod \"7850180b-94dc-4e7b-9323-b2ba77d1884d\" (UID: \"7850180b-94dc-4e7b-9323-b2ba77d1884d\") "
Oct 29 23:45:20.168592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.168033    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/984a1e35-8988-450a-80ad-76b3bfe60f38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984a1e35-8988-450a-80ad-76b3bfe60f38-plugins-config-volume\") pod \"984a1e35-8988-450a-80ad-76b3bfe60f38\" (UID: \"984a1e35-8988-450a-80ad-76b3bfe60f38\") "
Oct 29 23:45:20.169288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.169261    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7850180b-94dc-4e7b-9323-b2ba77d1884d-plugins-config-volume" pod "7850180b-94dc-4e7b-9323-b2ba77d1884d" (UID: "7850180b-94dc-4e7b-9323-b2ba77d1884d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:20.169881 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.169843    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7850180b-94dc-4e7b-9323-b2ba77d1884d-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:20.179062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.179025    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984a1e35-8988-450a-80ad-76b3bfe60f38-plugins-config-volume" pod "984a1e35-8988-450a-80ad-76b3bfe60f38" (UID: "984a1e35-8988-450a-80ad-76b3bfe60f38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:20.179224 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.179177    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984a1e35-8988-450a-80ad-76b3bfe60f38-kube-api-access-p5tvx" pod "984a1e35-8988-450a-80ad-76b3bfe60f38" (UID: "984a1e35-8988-450a-80ad-76b3bfe60f38"). InnerVolumeSpecName "kube-api-access-p5tvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:20.182981 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.182952    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7850180b-94dc-4e7b-9323-b2ba77d1884d-kube-api-access-qftgd" pod "7850180b-94dc-4e7b-9323-b2ba77d1884d" (UID: "7850180b-94dc-4e7b-9323-b2ba77d1884d"). InnerVolumeSpecName "kube-api-access-qftgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:20.271687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.271657    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5tvx\" (UniqueName: \"kubernetes.io/projected/984a1e35-8988-450a-80ad-76b3bfe60f38-kube-api-access-p5tvx\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:20.271687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.271686    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qftgd\" (UniqueName: \"kubernetes.io/projected/7850180b-94dc-4e7b-9323-b2ba77d1884d-kube-api-access-qftgd\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:20.271920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.271699    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984a1e35-8988-450a-80ad-76b3bfe60f38-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:20.620556 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.620519    5428 generic.go:356] "Generic (PLEG): container finished" podID="3ba2006e-ff86-47a3-8d81-c935530476e8" containerID="2ed91a1bda85086c338a20058c814917f802b47c107039ef2d75cc8975bac253" exitCode=60
Oct 29 23:45:20.621082 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.620594    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n" event={"ID":"3ba2006e-ff86-47a3-8d81-c935530476e8","Type":"ContainerDied","Data":"2ed91a1bda85086c338a20058c814917f802b47c107039ef2d75cc8975bac253"}
Oct 29 23:45:20.621082 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.620658    5428 scope.go:122] "RemoveContainer" containerID="32ff871c8f8ea8db743ff5e4ac90d5c9fcc05ab625310bf8f63592bab7311498"
Oct 29 23:45:20.621207 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.621160    5428 scope.go:122] "RemoveContainer" containerID="2ed91a1bda85086c338a20058c814917f802b47c107039ef2d75cc8975bac253"
Oct 29 23:45:20.621554 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:20.621526    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n_odh-ai-gateway-infra(3ba2006e-ff86-47a3-8d81-c935530476e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n" podUID="3ba2006e-ff86-47a3-8d81-c935530476e8"
Oct 29 23:45:20.627865 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.627839    5428 generic.go:356] "Generic (PLEG): container finished" podID="984a1e35-8988-450a-80ad-76b3bfe60f38" containerID="29f1b54421724b2a054c1c763dfbc1aa6546cec1ae05f8c3a1f3dfee3e0137f2" exitCode=0
Oct 29 23:45:20.627955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.627871    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m"
Oct 29 23:45:20.627955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.627910    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m" event={"ID":"984a1e35-8988-450a-80ad-76b3bfe60f38","Type":"ContainerDied","Data":"29f1b54421724b2a054c1c763dfbc1aa6546cec1ae05f8c3a1f3dfee3e0137f2"}
Oct 29 23:45:20.628062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.627956    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m" event={"ID":"984a1e35-8988-450a-80ad-76b3bfe60f38","Type":"ContainerDied","Data":"de29f328d98e5083daa19301b43bcf515705a5c557c3d542fc795f82073c2a38"}
Oct 29 23:45:20.632788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.632765    5428 generic.go:356] "Generic (PLEG): container finished" podID="918c3198-f2b2-413c-9002-91b324b0aeae" containerID="0eb9ae3314eb97f02f7316417516a56ec6239ca8dab5cef1d4ed9e7bc6f85ecd" exitCode=60
Oct 29 23:45:20.632908 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.632860    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75" event={"ID":"918c3198-f2b2-413c-9002-91b324b0aeae","Type":"ContainerDied","Data":"0eb9ae3314eb97f02f7316417516a56ec6239ca8dab5cef1d4ed9e7bc6f85ecd"}
Oct 29 23:45:20.633214 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.633197    5428 scope.go:122] "RemoveContainer" containerID="0eb9ae3314eb97f02f7316417516a56ec6239ca8dab5cef1d4ed9e7bc6f85ecd"
Oct 29 23:45:20.633503 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:20.633482    5428 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75_odh-ai-gateway-infra(918c3198-f2b2-413c-9002-91b324b0aeae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75" podUID="918c3198-f2b2-413c-9002-91b324b0aeae"
Oct 29 23:45:20.634263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.634234    5428 generic.go:356] "Generic (PLEG): container finished" podID="7850180b-94dc-4e7b-9323-b2ba77d1884d" containerID="45f9a74cc7fc6d2d7ee2ef8f9777dc104219d995d5878cfd28ee841542cacc34" exitCode=0
Oct 29 23:45:20.634381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.634282    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br"
Oct 29 23:45:20.634381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.634317    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br" event={"ID":"7850180b-94dc-4e7b-9323-b2ba77d1884d","Type":"ContainerDied","Data":"45f9a74cc7fc6d2d7ee2ef8f9777dc104219d995d5878cfd28ee841542cacc34"}
Oct 29 23:45:20.634381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.634356    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br" event={"ID":"7850180b-94dc-4e7b-9323-b2ba77d1884d","Type":"ContainerDied","Data":"af062bdb5e2cba869896edfed55bc2a429a041800e7b79bcce50a537b7fccce3"}
Oct 29 23:45:20.694882 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.694851    5428 scope.go:122] "RemoveContainer" containerID="29f1b54421724b2a054c1c763dfbc1aa6546cec1ae05f8c3a1f3dfee3e0137f2"
Oct 29 23:45:20.754580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.754555    5428 scope.go:122] "RemoveContainer" containerID="29f1b54421724b2a054c1c763dfbc1aa6546cec1ae05f8c3a1f3dfee3e0137f2"
Oct 29 23:45:20.755691 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:20.755562    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f1b54421724b2a054c1c763dfbc1aa6546cec1ae05f8c3a1f3dfee3e0137f2\": container with ID starting with 29f1b54421724b2a054c1c763dfbc1aa6546cec1ae05f8c3a1f3dfee3e0137f2 not found: ID does not exist" containerID="29f1b54421724b2a054c1c763dfbc1aa6546cec1ae05f8c3a1f3dfee3e0137f2"
Oct 29 23:45:20.755691 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.755600    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f1b54421724b2a054c1c763dfbc1aa6546cec1ae05f8c3a1f3dfee3e0137f2"} err="failed to get container status \"29f1b54421724b2a054c1c763dfbc1aa6546cec1ae05f8c3a1f3dfee3e0137f2\": rpc error: code = NotFound desc = could not find container \"29f1b54421724b2a054c1c763dfbc1aa6546cec1ae05f8c3a1f3dfee3e0137f2\": container with ID starting with 29f1b54421724b2a054c1c763dfbc1aa6546cec1ae05f8c3a1f3dfee3e0137f2 not found: ID does not exist"
Oct 29 23:45:20.755691 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.755628    5428 scope.go:122] "RemoveContainer" containerID="3a16cb3e916946d8ff10a637c917e1af061e8d1d02dc3d3752fd627d479c8841"
Oct 29 23:45:20.802342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.802310    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36d961cc-02de-4c2d-8f84-87e66cf01259" path="/var/lib/kubelet/pods/36d961cc-02de-4c2d-8f84-87e66cf01259/volumes"
Oct 29 23:45:20.803335 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.803310    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9553cf74-cf03-4ad2-8241-d82087a823f9" path="/var/lib/kubelet/pods/9553cf74-cf03-4ad2-8241-d82087a823f9/volumes"
Oct 29 23:45:20.804418 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.804398    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f1310de-26bc-46b9-86d6-10c8d1d49a93" path="/var/lib/kubelet/pods/9f1310de-26bc-46b9-86d6-10c8d1d49a93/volumes"
Oct 29 23:45:20.805137 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.805118    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a1fa0e-f264-46bb-83c0-efd1be480dd3" path="/var/lib/kubelet/pods/b0a1fa0e-f264-46bb-83c0-efd1be480dd3/volumes"
Oct 29 23:45:20.811528 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.811245    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b288d408-943e-4b45-abc9-7d942e5eb3f2" path="/var/lib/kubelet/pods/b288d408-943e-4b45-abc9-7d942e5eb3f2/volumes"
Oct 29 23:45:20.811717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.811700    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8fcfb61-2011-477e-9138-c7eb186665a6" path="/var/lib/kubelet/pods/d8fcfb61-2011-477e-9138-c7eb186665a6/volumes"
Oct 29 23:45:20.812167 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.812146    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br"]
Oct 29 23:45:20.812277 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.812182    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e10975-7b58b9b574-mr6br"]
Oct 29 23:45:20.813672 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.813653    5428 scope.go:122] "RemoveContainer" containerID="45f9a74cc7fc6d2d7ee2ef8f9777dc104219d995d5878cfd28ee841542cacc34"
Oct 29 23:45:20.837274 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.837245    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4"]
Oct 29 23:45:20.850233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.850212    5428 scope.go:122] "RemoveContainer" containerID="45f9a74cc7fc6d2d7ee2ef8f9777dc104219d995d5878cfd28ee841542cacc34"
Oct 29 23:45:20.851506 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:20.851013    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45f9a74cc7fc6d2d7ee2ef8f9777dc104219d995d5878cfd28ee841542cacc34\": container with ID starting with 45f9a74cc7fc6d2d7ee2ef8f9777dc104219d995d5878cfd28ee841542cacc34 not found: ID does not exist" containerID="45f9a74cc7fc6d2d7ee2ef8f9777dc104219d995d5878cfd28ee841542cacc34"
Oct 29 23:45:20.851506 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.851050    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45f9a74cc7fc6d2d7ee2ef8f9777dc104219d995d5878cfd28ee841542cacc34"} err="failed to get container status \"45f9a74cc7fc6d2d7ee2ef8f9777dc104219d995d5878cfd28ee841542cacc34\": rpc error: code = NotFound desc = could not find container \"45f9a74cc7fc6d2d7ee2ef8f9777dc104219d995d5878cfd28ee841542cacc34\": container with ID starting with 45f9a74cc7fc6d2d7ee2ef8f9777dc104219d995d5878cfd28ee841542cacc34 not found: ID does not exist"
Oct 29 23:45:20.922820 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.912340    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m"]
Oct 29 23:45:20.923136 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:20.922917    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e10975-654ddf6d7f-fwf2m"]
Oct 29 23:45:21.004432 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.004295    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc"
Oct 29 23:45:21.014180 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.014125    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk"
Oct 29 23:45:21.089134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.089045    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc6e9365-f21c-49b3-b513-c17c1113cb41-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc6e9365-f21c-49b3-b513-c17c1113cb41-maas-api-service-ca\") pod \"fc6e9365-f21c-49b3-b513-c17c1113cb41\" (UID: \"fc6e9365-f21c-49b3-b513-c17c1113cb41\") "
Oct 29 23:45:21.089134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.089133    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ffbf666-ed06-4d62-bf9a-d1786bcf071f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffbf666-ed06-4d62-bf9a-d1786bcf071f-maas-api-service-ca\") pod \"9ffbf666-ed06-4d62-bf9a-d1786bcf071f\" (UID: \"9ffbf666-ed06-4d62-bf9a-d1786bcf071f\") "
Oct 29 23:45:21.090135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.090077    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc6e9365-f21c-49b3-b513-c17c1113cb41-maas-api-service-ca" pod "fc6e9365-f21c-49b3-b513-c17c1113cb41" (UID: "fc6e9365-f21c-49b3-b513-c17c1113cb41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:21.090386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.090323    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc6e9365-f21c-49b3-b513-c17c1113cb41-kube-api-access-j2bzc\" (UniqueName: \"kubernetes.io/projected/fc6e9365-f21c-49b3-b513-c17c1113cb41-kube-api-access-j2bzc\") pod \"fc6e9365-f21c-49b3-b513-c17c1113cb41\" (UID: \"fc6e9365-f21c-49b3-b513-c17c1113cb41\") "
Oct 29 23:45:21.090548 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.090524    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ffbf666-ed06-4d62-bf9a-d1786bcf071f-kube-api-access-7t4qh\" (UniqueName: \"kubernetes.io/projected/9ffbf666-ed06-4d62-bf9a-d1786bcf071f-kube-api-access-7t4qh\") pod \"9ffbf666-ed06-4d62-bf9a-d1786bcf071f\" (UID: \"9ffbf666-ed06-4d62-bf9a-d1786bcf071f\") "
Oct 29 23:45:21.091572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.091230    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc6e9365-f21c-49b3-b513-c17c1113cb41-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:21.093221 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.093194    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ffbf666-ed06-4d62-bf9a-d1786bcf071f-maas-api-service-ca" pod "9ffbf666-ed06-4d62-bf9a-d1786bcf071f" (UID: "9ffbf666-ed06-4d62-bf9a-d1786bcf071f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:21.095695 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.095667    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ffbf666-ed06-4d62-bf9a-d1786bcf071f-kube-api-access-7t4qh" pod "9ffbf666-ed06-4d62-bf9a-d1786bcf071f" (UID: "9ffbf666-ed06-4d62-bf9a-d1786bcf071f"). InnerVolumeSpecName "kube-api-access-7t4qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:21.108330 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.108152    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc6e9365-f21c-49b3-b513-c17c1113cb41-kube-api-access-j2bzc" pod "fc6e9365-f21c-49b3-b513-c17c1113cb41" (UID: "fc6e9365-f21c-49b3-b513-c17c1113cb41"). InnerVolumeSpecName "kube-api-access-j2bzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:21.154379 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.154351    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4"
Oct 29 23:45:21.192383 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.192350    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ffbf666-ed06-4d62-bf9a-d1786bcf071f-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:21.192383 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.192378    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2bzc\" (UniqueName: \"kubernetes.io/projected/fc6e9365-f21c-49b3-b513-c17c1113cb41-kube-api-access-j2bzc\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:21.192383 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.192388    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t4qh\" (UniqueName: \"kubernetes.io/projected/9ffbf666-ed06-4d62-bf9a-d1786bcf071f-kube-api-access-7t4qh\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:21.293771 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.293691    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ff49c0b-5f9d-411a-a767-77897583a7a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff49c0b-5f9d-411a-a767-77897583a7a3-maas-api-service-ca\") pod \"4ff49c0b-5f9d-411a-a767-77897583a7a3\" (UID: \"4ff49c0b-5f9d-411a-a767-77897583a7a3\") "
Oct 29 23:45:21.293997 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.293871    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ff49c0b-5f9d-411a-a767-77897583a7a3-kube-api-access-7lncc\" (UniqueName: \"kubernetes.io/projected/4ff49c0b-5f9d-411a-a767-77897583a7a3-kube-api-access-7lncc\") pod \"4ff49c0b-5f9d-411a-a767-77897583a7a3\" (UID: \"4ff49c0b-5f9d-411a-a767-77897583a7a3\") "
Oct 29 23:45:21.294098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.294074    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ff49c0b-5f9d-411a-a767-77897583a7a3-maas-api-service-ca" pod "4ff49c0b-5f9d-411a-a767-77897583a7a3" (UID: "4ff49c0b-5f9d-411a-a767-77897583a7a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:21.294665 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.294643    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff49c0b-5f9d-411a-a767-77897583a7a3-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:21.296586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.296531    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ff49c0b-5f9d-411a-a767-77897583a7a3-kube-api-access-7lncc" pod "4ff49c0b-5f9d-411a-a767-77897583a7a3" (UID: "4ff49c0b-5f9d-411a-a767-77897583a7a3"). InnerVolumeSpecName "kube-api-access-7lncc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:21.396682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.396641    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lncc\" (UniqueName: \"kubernetes.io/projected/4ff49c0b-5f9d-411a-a767-77897583a7a3-kube-api-access-7lncc\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:21.648597 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.648570    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4"
Oct 29 23:45:21.649121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.648604    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4" event={"ID":"4ff49c0b-5f9d-411a-a767-77897583a7a3","Type":"ContainerDied","Data":"0ceb70d32f9f496c56c8e8f5d2a982bb2c2047b25658406c5f71367f12397373"}
Oct 29 23:45:21.649121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.648677    5428 scope.go:122] "RemoveContainer" containerID="8234ac5cc0e01ccac16e0b6a17f17c1697aabad1e103b04d98205368c7121404"
Oct 29 23:45:21.650420 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.650357    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk" event={"ID":"fc6e9365-f21c-49b3-b513-c17c1113cb41","Type":"ContainerDied","Data":"bce97047fd5a07e3973a6db0b8c3ea7b0261d7089bfed0d73ce268225e606116"}
Oct 29 23:45:21.650420 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.650376    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk"
Oct 29 23:45:21.653528 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.653496    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc"
Oct 29 23:45:21.653646 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.653529    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc" event={"ID":"9ffbf666-ed06-4d62-bf9a-d1786bcf071f","Type":"ContainerDied","Data":"cdb28d41e60cea6bb214d278b50758b820daca585dcc6f1d32cfb934dc41d7c9"}
Oct 29 23:45:21.664251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.663240    5428 scope.go:122] "RemoveContainer" containerID="463fac0662eed54a92b6535d84884c63237e705ef594ab050fdd2978c427ff4c"
Oct 29 23:45:21.677375 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.677251    5428 scope.go:122] "RemoveContainer" containerID="036789793a2decf04b33658808d7b71acf87854ef96192bfcd29cbe32b7a917b"
Oct 29 23:45:21.680535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.680511    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n"]
Oct 29 23:45:21.743082 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.743018    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:21.760051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.749782    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc"]
Oct 29 23:45:21.760051 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.758392    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-99dfb0-29888625-h2vrc"]
Oct 29 23:45:21.764257 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.764234    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:21.781649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.781617    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk"]
Oct 29 23:45:21.808857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.808822    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2077b4-29888625-mhbpk"]
Oct 29 23:45:21.884945 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.884912    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75"]
Oct 29 23:45:21.921159 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.910008    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4"]
Oct 29 23:45:21.921159 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.916555    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2126fb-29888625-7dlb4"]
Oct 29 23:45:21.939733 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.924071    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:21.991142 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:21.991029    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n"
Oct 29 23:45:22.123124 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.123084    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba2006e-ff86-47a3-8d81-c935530476e8-kube-api-access-5mh6d\" (UniqueName: \"kubernetes.io/projected/3ba2006e-ff86-47a3-8d81-c935530476e8-kube-api-access-5mh6d\") pod \"3ba2006e-ff86-47a3-8d81-c935530476e8\" (UID: \"3ba2006e-ff86-47a3-8d81-c935530476e8\") "
Oct 29 23:45:22.123311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.123130    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ba2006e-ff86-47a3-8d81-c935530476e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ba2006e-ff86-47a3-8d81-c935530476e8-maas-api-service-ca\") pod \"3ba2006e-ff86-47a3-8d81-c935530476e8\" (UID: \"3ba2006e-ff86-47a3-8d81-c935530476e8\") "
Oct 29 23:45:22.123874 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.123826    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba2006e-ff86-47a3-8d81-c935530476e8-maas-api-service-ca" pod "3ba2006e-ff86-47a3-8d81-c935530476e8" (UID: "3ba2006e-ff86-47a3-8d81-c935530476e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:22.133830 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.124281    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ba2006e-ff86-47a3-8d81-c935530476e8-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:22.155399 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.155370    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba2006e-ff86-47a3-8d81-c935530476e8-kube-api-access-5mh6d" pod "3ba2006e-ff86-47a3-8d81-c935530476e8" (UID: "3ba2006e-ff86-47a3-8d81-c935530476e8"). InnerVolumeSpecName "kube-api-access-5mh6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:22.190583 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.190524    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75"
Oct 29 23:45:22.226576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.226545    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mh6d\" (UniqueName: \"kubernetes.io/projected/3ba2006e-ff86-47a3-8d81-c935530476e8-kube-api-access-5mh6d\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:22.328059 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.328025    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/918c3198-f2b2-413c-9002-91b324b0aeae-kube-api-access-ctr7j\" (UniqueName: \"kubernetes.io/projected/918c3198-f2b2-413c-9002-91b324b0aeae-kube-api-access-ctr7j\") pod \"918c3198-f2b2-413c-9002-91b324b0aeae\" (UID: \"918c3198-f2b2-413c-9002-91b324b0aeae\") "
Oct 29 23:45:22.328290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.328268    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/918c3198-f2b2-413c-9002-91b324b0aeae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/918c3198-f2b2-413c-9002-91b324b0aeae-maas-api-service-ca\") pod \"918c3198-f2b2-413c-9002-91b324b0aeae\" (UID: \"918c3198-f2b2-413c-9002-91b324b0aeae\") "
Oct 29 23:45:22.328795 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.328764    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918c3198-f2b2-413c-9002-91b324b0aeae-maas-api-service-ca" pod "918c3198-f2b2-413c-9002-91b324b0aeae" (UID: "918c3198-f2b2-413c-9002-91b324b0aeae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:22.329029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.329005    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/918c3198-f2b2-413c-9002-91b324b0aeae-maas-api-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:22.344519 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.344488    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918c3198-f2b2-413c-9002-91b324b0aeae-kube-api-access-ctr7j" pod "918c3198-f2b2-413c-9002-91b324b0aeae" (UID: "918c3198-f2b2-413c-9002-91b324b0aeae"). InnerVolumeSpecName "kube-api-access-ctr7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:22.430549 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.430517    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctr7j\" (UniqueName: \"kubernetes.io/projected/918c3198-f2b2-413c-9002-91b324b0aeae-kube-api-access-ctr7j\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:22.671322 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.671290    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75"
Oct 29 23:45:22.671859 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.671299    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75" event={"ID":"918c3198-f2b2-413c-9002-91b324b0aeae","Type":"ContainerDied","Data":"2a53ece70a2f498e32f99bf0f9087a210615ac91165b10b872bdcbf7208ca50d"}
Oct 29 23:45:22.671859 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.671451    5428 scope.go:122] "RemoveContainer" containerID="0eb9ae3314eb97f02f7316417516a56ec6239ca8dab5cef1d4ed9e7bc6f85ecd"
Oct 29 23:45:22.673181 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.673144    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n" event={"ID":"3ba2006e-ff86-47a3-8d81-c935530476e8","Type":"ContainerDied","Data":"2104a31b364b9c455b8d381d37f81930f78b38358f4495921a074e704b409a64"}
Oct 29 23:45:22.673181 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.673168    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n"
Oct 29 23:45:22.685376 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.685358    5428 scope.go:122] "RemoveContainer" containerID="2ed91a1bda85086c338a20058c814917f802b47c107039ef2d75cc8975bac253"
Oct 29 23:45:22.689687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.689667    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:45:22.730741 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.730287    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n"]
Oct 29 23:45:22.739542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.739512    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-26b322-29888625-fzs6n"]
Oct 29 23:45:22.768228 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.768196    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75"]
Oct 29 23:45:22.784981 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.784953    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ba2006e-ff86-47a3-8d81-c935530476e8" path="/var/lib/kubelet/pods/3ba2006e-ff86-47a3-8d81-c935530476e8/volumes"
Oct 29 23:45:22.785611 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.785594    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ff49c0b-5f9d-411a-a767-77897583a7a3" path="/var/lib/kubelet/pods/4ff49c0b-5f9d-411a-a767-77897583a7a3/volumes"
Oct 29 23:45:22.786254 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.786209    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7850180b-94dc-4e7b-9323-b2ba77d1884d" path="/var/lib/kubelet/pods/7850180b-94dc-4e7b-9323-b2ba77d1884d/volumes"
Oct 29 23:45:22.787229 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.787211    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="984a1e35-8988-450a-80ad-76b3bfe60f38" path="/var/lib/kubelet/pods/984a1e35-8988-450a-80ad-76b3bfe60f38/volumes"
Oct 29 23:45:22.787769 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.787753    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ffbf666-ed06-4d62-bf9a-d1786bcf071f" path="/var/lib/kubelet/pods/9ffbf666-ed06-4d62-bf9a-d1786bcf071f/volumes"
Oct 29 23:45:22.800295 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.800274    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc6e9365-f21c-49b3-b513-c17c1113cb41" path="/var/lib/kubelet/pods/fc6e9365-f21c-49b3-b513-c17c1113cb41/volumes"
Oct 29 23:45:22.800775 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:22.800758    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-52f67b-29888625-qzp75"]
Oct 29 23:45:23.872154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:23.872116    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:24.780927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:24.780858    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="918c3198-f2b2-413c-9002-91b324b0aeae" path="/var/lib/kubelet/pods/918c3198-f2b2-413c-9002-91b324b0aeae/volumes"
Oct 29 23:45:25.201948 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:25.201916    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:25.323350 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:25.323322    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:25.539873 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:25.539773    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:25.621991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:25.621951    5428 patch_prober.go:29] interesting pod/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:45:25.622172 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:25.622021    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp" podUID="27603afe-5712-43b2-aaed-056d472de92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:45:25.730012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:25.729968    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:26.905938 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:26.905897    5428 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:45:26.906492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:26.905953    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2" podUID="772a13d1-e842-4cf0-b36e-09e07d7936e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:45:27.729861 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:27.729829    5428 generic.go:356] "Generic (PLEG): container finished" podID="27603afe-5712-43b2-aaed-056d472de92a" containerID="d4637797b8d50243bb72281afd00d1ca22f09fcb4e67ceda6cc7252bbd1c49f7" exitCode=0
Oct 29 23:45:27.730039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:27.729912    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp" event={"ID":"27603afe-5712-43b2-aaed-056d472de92a","Type":"ContainerDied","Data":"d4637797b8d50243bb72281afd00d1ca22f09fcb4e67ceda6cc7252bbd1c49f7"}
Oct 29 23:45:27.730522 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:27.730498    5428 scope.go:122] "RemoveContainer" containerID="d4637797b8d50243bb72281afd00d1ca22f09fcb4e67ceda6cc7252bbd1c49f7"
Oct 29 23:45:27.755874 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:27.732812    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:45:27.755874 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:27.732908    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:45:27.755874 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:27.732959    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:45:27.768144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:27.768115    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:28.746300 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:28.746263    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp" event={"ID":"27603afe-5712-43b2-aaed-056d472de92a","Type":"ContainerStarted","Data":"0200d03b41b9ad520c31f614906adb40ce05e2382eb9ca23711e934ce154513b"}
Oct 29 23:45:29.024162 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:29.024089    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:45:29.024422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:29.024202    5428 patch_prober.go:29] interesting pod/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 29 23:45:29.024422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:29.024249    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp" podUID="27603afe-5712-43b2-aaed-056d472de92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 29 23:45:29.282778 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:29.282699    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:29.770824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:29.767702    5428 generic.go:356] "Generic (PLEG): container finished" podID="772a13d1-e842-4cf0-b36e-09e07d7936e4" containerID="aee97831fe33e6a456d2c3de5ad3b8878f14f57135fe327039c1d05ee651d359" exitCode=0
Oct 29 23:45:29.770824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:29.768735    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2" event={"ID":"772a13d1-e842-4cf0-b36e-09e07d7936e4","Type":"ContainerDied","Data":"aee97831fe33e6a456d2c3de5ad3b8878f14f57135fe327039c1d05ee651d359"}
Oct 29 23:45:29.770824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:29.769244    5428 scope.go:122] "RemoveContainer" containerID="aee97831fe33e6a456d2c3de5ad3b8878f14f57135fe327039c1d05ee651d359"
Oct 29 23:45:29.790831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:29.783039    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:45:29.790831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:29.783153    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:45:29.790831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:29.783202    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:45:30.024385 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.024301    5428 patch_prober.go:29] interesting pod/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 29 23:45:30.024385 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.024351    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp" podUID="27603afe-5712-43b2-aaed-056d472de92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 29 23:45:30.224841 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.224769    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"]
Oct 29 23:45:30.225776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225751    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b0a1fa0e-f264-46bb-83c0-efd1be480dd3" containerName="cleanup"
Oct 29 23:45:30.225949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225779    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a1fa0e-f264-46bb-83c0-efd1be480dd3" containerName="cleanup"
Oct 29 23:45:30.225949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225796    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9ffbf666-ed06-4d62-bf9a-d1786bcf071f" containerName="cleanup"
Oct 29 23:45:30.225949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225820    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ffbf666-ed06-4d62-bf9a-d1786bcf071f" containerName="cleanup"
Oct 29 23:45:30.225949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225842    5428 cpu_manager.go:414] "containerMap: removing container" podUID="d8fcfb61-2011-477e-9138-c7eb186665a6" containerName="cleanup"
Oct 29 23:45:30.225949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225849    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8fcfb61-2011-477e-9138-c7eb186665a6" containerName="cleanup"
Oct 29 23:45:30.225949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225858    5428 cpu_manager.go:414] "containerMap: removing container" podUID="918c3198-f2b2-413c-9002-91b324b0aeae" containerName="cleanup"
Oct 29 23:45:30.225949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225865    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918c3198-f2b2-413c-9002-91b324b0aeae" containerName="cleanup"
Oct 29 23:45:30.225949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225874    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b0a1fa0e-f264-46bb-83c0-efd1be480dd3" containerName="cleanup"
Oct 29 23:45:30.225949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225881    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a1fa0e-f264-46bb-83c0-efd1be480dd3" containerName="cleanup"
Oct 29 23:45:30.225949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225897    5428 cpu_manager.go:414] "containerMap: removing container" podUID="d8fcfb61-2011-477e-9138-c7eb186665a6" containerName="cleanup"
Oct 29 23:45:30.225949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225904    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8fcfb61-2011-477e-9138-c7eb186665a6" containerName="cleanup"
Oct 29 23:45:30.225949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225913    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b288d408-943e-4b45-abc9-7d942e5eb3f2" containerName="maas-api"
Oct 29 23:45:30.225949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225920    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b288d408-943e-4b45-abc9-7d942e5eb3f2" containerName="maas-api"
Oct 29 23:45:30.225949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225951    5428 cpu_manager.go:414] "containerMap: removing container" podUID="3ba2006e-ff86-47a3-8d81-c935530476e8" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225959    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ba2006e-ff86-47a3-8d81-c935530476e8" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225978    5428 cpu_manager.go:414] "containerMap: removing container" podUID="fc6e9365-f21c-49b3-b513-c17c1113cb41" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225986    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc6e9365-f21c-49b3-b513-c17c1113cb41" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.225997    5428 cpu_manager.go:414] "containerMap: removing container" podUID="4ff49c0b-5f9d-411a-a767-77897583a7a3" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226005    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ff49c0b-5f9d-411a-a767-77897583a7a3" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226017    5428 cpu_manager.go:414] "containerMap: removing container" podUID="4ff49c0b-5f9d-411a-a767-77897583a7a3" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226024    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ff49c0b-5f9d-411a-a767-77897583a7a3" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226038    5428 cpu_manager.go:414] "containerMap: removing container" podUID="36d961cc-02de-4c2d-8f84-87e66cf01259" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226045    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d961cc-02de-4c2d-8f84-87e66cf01259" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226057    5428 cpu_manager.go:414] "containerMap: removing container" podUID="fc6e9365-f21c-49b3-b513-c17c1113cb41" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226067    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc6e9365-f21c-49b3-b513-c17c1113cb41" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226087    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9553cf74-cf03-4ad2-8241-d82087a823f9" containerName="payload-pre-processing"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226095    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9553cf74-cf03-4ad2-8241-d82087a823f9" containerName="payload-pre-processing"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226112    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9f1310de-26bc-46b9-86d6-10c8d1d49a93" containerName="payload-processing"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226120    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f1310de-26bc-46b9-86d6-10c8d1d49a93" containerName="payload-processing"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226133    5428 cpu_manager.go:414] "containerMap: removing container" podUID="984a1e35-8988-450a-80ad-76b3bfe60f38" containerName="payload-pre-processing"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226141    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="984a1e35-8988-450a-80ad-76b3bfe60f38" containerName="payload-pre-processing"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226154    5428 cpu_manager.go:414] "containerMap: removing container" podUID="7850180b-94dc-4e7b-9323-b2ba77d1884d" containerName="payload-processing"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226162    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7850180b-94dc-4e7b-9323-b2ba77d1884d" containerName="payload-processing"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226174    5428 cpu_manager.go:414] "containerMap: removing container" podUID="fc6e9365-f21c-49b3-b513-c17c1113cb41" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226181    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc6e9365-f21c-49b3-b513-c17c1113cb41" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226190    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9ffbf666-ed06-4d62-bf9a-d1786bcf071f" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226197    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ffbf666-ed06-4d62-bf9a-d1786bcf071f" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226221    5428 cpu_manager.go:414] "containerMap: removing container" podUID="36d961cc-02de-4c2d-8f84-87e66cf01259" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226228    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d961cc-02de-4c2d-8f84-87e66cf01259" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226266    5428 cpu_manager.go:414] "containerMap: removing container" podUID="3ba2006e-ff86-47a3-8d81-c935530476e8" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226274    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ba2006e-ff86-47a3-8d81-c935530476e8" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226286    5428 cpu_manager.go:414] "containerMap: removing container" podUID="918c3198-f2b2-413c-9002-91b324b0aeae" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226294    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918c3198-f2b2-413c-9002-91b324b0aeae" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226525    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ff49c0b-5f9d-411a-a767-77897583a7a3" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226540    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d961cc-02de-4c2d-8f84-87e66cf01259" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226548    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8fcfb61-2011-477e-9138-c7eb186665a6" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226556    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc6e9365-f21c-49b3-b513-c17c1113cb41" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226570    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9553cf74-cf03-4ad2-8241-d82087a823f9" containerName="payload-pre-processing"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226581    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f1310de-26bc-46b9-86d6-10c8d1d49a93" containerName="payload-processing"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226591    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="984a1e35-8988-450a-80ad-76b3bfe60f38" containerName="payload-pre-processing"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226601    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="7850180b-94dc-4e7b-9323-b2ba77d1884d" containerName="payload-processing"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226608    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc6e9365-f21c-49b3-b513-c17c1113cb41" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226615    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ffbf666-ed06-4d62-bf9a-d1786bcf071f" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226627    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d961cc-02de-4c2d-8f84-87e66cf01259" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226642    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ba2006e-ff86-47a3-8d81-c935530476e8" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226651    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ba2006e-ff86-47a3-8d81-c935530476e8" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226660    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ffbf666-ed06-4d62-bf9a-d1786bcf071f" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226669    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="918c3198-f2b2-413c-9002-91b324b0aeae" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226678    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="918c3198-f2b2-413c-9002-91b324b0aeae" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226686    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a1fa0e-f264-46bb-83c0-efd1be480dd3" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226695    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ff49c0b-5f9d-411a-a767-77897583a7a3" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226705    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a1fa0e-f264-46bb-83c0-efd1be480dd3" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226714    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ffbf666-ed06-4d62-bf9a-d1786bcf071f" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226725    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8fcfb61-2011-477e-9138-c7eb186665a6" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226734    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="918c3198-f2b2-413c-9002-91b324b0aeae" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226741    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a1fa0e-f264-46bb-83c0-efd1be480dd3" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226753    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8fcfb61-2011-477e-9138-c7eb186665a6" containerName="cleanup"
Oct 29 23:45:30.227064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.226761    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b288d408-943e-4b45-abc9-7d942e5eb3f2" containerName="maas-api"
Oct 29 23:45:30.231450 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.231098    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"]
Oct 29 23:45:30.231450 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.231201    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.253901 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.253859    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-825472\"/\"default-dockercfg-8w7p7\""
Oct 29 23:45:30.254055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.253905    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-825472\"/\"test-model-825472-kserve-self-signed-certs\""
Oct 29 23:45:30.257275 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.256765    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-home\") pod \"test-model-825472-kserve-767979fd87-dghdj\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") " pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.257275 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.256841    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-model-cache\") pod \"test-model-825472-kserve-767979fd87-dghdj\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") " pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.257275 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.256883    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djh9x\" (UniqueName: \"kubernetes.io/projected/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-kube-api-access-djh9x\") pod \"test-model-825472-kserve-767979fd87-dghdj\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") " pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.257275 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.256919    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-tmp-dir\") pod \"test-model-825472-kserve-767979fd87-dghdj\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") " pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.257275 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.256963    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-dshm\") pod \"test-model-825472-kserve-767979fd87-dghdj\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") " pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.257275 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.257115    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-tls-certs\") pod \"test-model-825472-kserve-767979fd87-dghdj\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") " pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.258527 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.258193    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-825472\"/\"kube-root-ca.crt\""
Oct 29 23:45:30.258527 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.258345    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-825472\"/\"openshift-service-ca.crt\""
Oct 29 23:45:30.291473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.291346    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:30.358532 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.358496    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-home\") pod \"test-model-825472-kserve-767979fd87-dghdj\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") " pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.358762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.358548    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-model-cache\") pod \"test-model-825472-kserve-767979fd87-dghdj\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") " pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.358762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.358598    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djh9x\" (UniqueName: \"kubernetes.io/projected/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-kube-api-access-djh9x\") pod \"test-model-825472-kserve-767979fd87-dghdj\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") " pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.358762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.358638    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-tmp-dir\") pod \"test-model-825472-kserve-767979fd87-dghdj\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") " pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.358987 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.358848    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-dshm\") pod \"test-model-825472-kserve-767979fd87-dghdj\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") " pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.359041 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.358980    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-home\") pod \"test-model-825472-kserve-767979fd87-dghdj\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") " pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.359041 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.358999    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-model-cache\") pod \"test-model-825472-kserve-767979fd87-dghdj\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") " pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.359146 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.359121    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-tls-certs\") pod \"test-model-825472-kserve-767979fd87-dghdj\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") " pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.381825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.379402    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-tmp-dir\") pod \"test-model-825472-kserve-767979fd87-dghdj\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") " pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.392982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.392941    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-dshm\") pod \"test-model-825472-kserve-767979fd87-dghdj\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") " pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.394185 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.394144    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-tls-certs\") pod \"test-model-825472-kserve-767979fd87-dghdj\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") " pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.394785 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.394514    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djh9x\" (UniqueName: \"kubernetes.io/projected/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-kube-api-access-djh9x\") pod \"test-model-825472-kserve-767979fd87-dghdj\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") " pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.418038 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.418000    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:30.549454 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.549364    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:30.580839 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.579604    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:30.809900 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.809813    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2" event={"ID":"772a13d1-e842-4cf0-b36e-09e07d7936e4","Type":"ContainerStarted","Data":"dc89684e7e7619f1c137f31d6f861fc3db539886352a9ee2f2f431f0522cb77d"}
Oct 29 23:45:30.950050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.950008    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:30.990161 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:30.987116    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"]
Oct 29 23:45:30.992996 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:45:30.992950    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5fbdcde_d3a1_4ebb_8f9b_b1ed8fcfa47a.slice/crio-037414b7e4e775a2b5aaa8c5fe198c9cd6e9d4c48a9ef627efda5aabf86045c1 WatchSource:0}: Error finding container 037414b7e4e775a2b5aaa8c5fe198c9cd6e9d4c48a9ef627efda5aabf86045c1: Status 404 returned error can't find the container with id 037414b7e4e775a2b5aaa8c5fe198c9cd6e9d4c48a9ef627efda5aabf86045c1
Oct 29 23:45:31.025596 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:31.025228    5428 patch_prober.go:29] interesting pod/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 29 23:45:31.025596 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:31.025278    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp" podUID="27603afe-5712-43b2-aaed-056d472de92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 29 23:45:31.384888 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:31.364116    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:45:31.384888 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:31.364269    5428 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 29 23:45:31.384888 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:31.364320    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2" podUID="772a13d1-e842-4cf0-b36e-09e07d7936e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 29 23:45:31.824606 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:31.824522    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj" event={"ID":"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a","Type":"ContainerStarted","Data":"f060cb872bf111e1b3738d92bceec2cd5679aadc6f58d5830a8c260495a47dc2"}
Oct 29 23:45:31.824606 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:31.824563    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj" event={"ID":"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a","Type":"ContainerStarted","Data":"037414b7e4e775a2b5aaa8c5fe198c9cd6e9d4c48a9ef627efda5aabf86045c1"}
Oct 29 23:45:31.912823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:31.911683    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj" podStartSLOduration=1.911658764 podStartE2EDuration="1.911658764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:31.882486976 +0000 UTC m=+3735.225435725" watchObservedRunningTime="2026-10-29 23:45:31.911658764 +0000 UTC m=+3735.254607509"
Oct 29 23:45:32.025331 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:32.025290    5428 patch_prober.go:29] interesting pod/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 29 23:45:32.025648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:32.025354    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp" podUID="27603afe-5712-43b2-aaed-056d472de92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 29 23:45:32.382826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:32.369272    5428 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 29 23:45:32.382826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:32.369332    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2" podUID="772a13d1-e842-4cf0-b36e-09e07d7936e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 29 23:45:32.549514 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:32.549475    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:33.024245 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:33.024191    5428 patch_prober.go:29] interesting pod/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 29 23:45:33.024659 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:33.024266    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp" podUID="27603afe-5712-43b2-aaed-056d472de92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 29 23:45:33.035816 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:33.035778    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:33.157268 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:33.157235    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:33.374826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:33.371549    5428 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 29 23:45:33.374826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:33.371605    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2" podUID="772a13d1-e842-4cf0-b36e-09e07d7936e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 29 23:45:33.837662 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:33.837574    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:34.025477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:34.024313    5428 patch_prober.go:29] interesting pod/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 29 23:45:34.025477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:34.024360    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp" podUID="27603afe-5712-43b2-aaed-056d472de92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 29 23:45:34.376829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:34.373221    5428 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 29 23:45:34.376829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:34.373277    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2" podUID="772a13d1-e842-4cf0-b36e-09e07d7936e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 29 23:45:35.024676 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:35.024634    5428 patch_prober.go:29] interesting pod/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 29 23:45:35.024972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:35.024687    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp" podUID="27603afe-5712-43b2-aaed-056d472de92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 29 23:45:35.339744 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:35.339619    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:35.382824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:35.374879    5428 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 29 23:45:35.382824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:35.374952    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2" podUID="772a13d1-e842-4cf0-b36e-09e07d7936e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 29 23:45:36.030666 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:36.030637    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:45:36.031080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:36.031057    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:45:36.031729 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:36.031710    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:45:36.384827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:36.383247    5428 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 29 23:45:36.384827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:36.383327    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2" podUID="772a13d1-e842-4cf0-b36e-09e07d7936e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 29 23:45:36.668896 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:36.668789    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5"]
Oct 29 23:45:36.801074 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:36.801037    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:36.916538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:36.916507    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:37.130272 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.130238    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"]
Oct 29 23:45:37.130509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.130485    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d" podUID="f38ef730-877a-49cd-98c2-64cbb7894032" containerName="maas-api" containerID="cri-o://f5393fcbecf5d6b18b53ec49053e19d9313dbab81a6f69e343572f7c614eb30e" gracePeriod=30
Oct 29 23:45:37.161972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.161951    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:37.286943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.286911    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:37.370976 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.370944    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:45:37.372224 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.371533    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:45:37.372571 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.372548    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:45:37.494665 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.494637    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"
Oct 29 23:45:37.537549 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.536246    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:37.649081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.649000    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f38ef730-877a-49cd-98c2-64cbb7894032-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38ef730-877a-49cd-98c2-64cbb7894032-openshift-service-ca\") pod \"f38ef730-877a-49cd-98c2-64cbb7894032\" (UID: \"f38ef730-877a-49cd-98c2-64cbb7894032\") "
Oct 29 23:45:37.649081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.649072    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f38ef730-877a-49cd-98c2-64cbb7894032-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38ef730-877a-49cd-98c2-64cbb7894032-maas-api-tls\") pod \"f38ef730-877a-49cd-98c2-64cbb7894032\" (UID: \"f38ef730-877a-49cd-98c2-64cbb7894032\") "
Oct 29 23:45:37.649272 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.649200    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f38ef730-877a-49cd-98c2-64cbb7894032-kube-api-access-fk6t9\" (UniqueName: \"kubernetes.io/projected/f38ef730-877a-49cd-98c2-64cbb7894032-kube-api-access-fk6t9\") pod \"f38ef730-877a-49cd-98c2-64cbb7894032\" (UID: \"f38ef730-877a-49cd-98c2-64cbb7894032\") "
Oct 29 23:45:37.649466 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.649436    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38ef730-877a-49cd-98c2-64cbb7894032-openshift-service-ca" pod "f38ef730-877a-49cd-98c2-64cbb7894032" (UID: "f38ef730-877a-49cd-98c2-64cbb7894032"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:37.649615 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.649596    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f38ef730-877a-49cd-98c2-64cbb7894032-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38ef730-877a-49cd-98c2-64cbb7894032-metrics-certs\") pod \"f38ef730-877a-49cd-98c2-64cbb7894032\" (UID: \"f38ef730-877a-49cd-98c2-64cbb7894032\") "
Oct 29 23:45:37.651944 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.651863    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38ef730-877a-49cd-98c2-64cbb7894032-kube-api-access-fk6t9" pod "f38ef730-877a-49cd-98c2-64cbb7894032" (UID: "f38ef730-877a-49cd-98c2-64cbb7894032"). InnerVolumeSpecName "kube-api-access-fk6t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:37.652279 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.652216    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f38ef730-877a-49cd-98c2-64cbb7894032-metrics-certs" pod "f38ef730-877a-49cd-98c2-64cbb7894032" (UID: "f38ef730-877a-49cd-98c2-64cbb7894032"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:45:37.652441 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.652392    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk6t9\" (UniqueName: \"kubernetes.io/projected/f38ef730-877a-49cd-98c2-64cbb7894032-kube-api-access-fk6t9\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:37.652441 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.652416    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38ef730-877a-49cd-98c2-64cbb7894032-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:37.656840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.653937    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f38ef730-877a-49cd-98c2-64cbb7894032-maas-api-tls" pod "f38ef730-877a-49cd-98c2-64cbb7894032" (UID: "f38ef730-877a-49cd-98c2-64cbb7894032"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:45:37.755067 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.755018    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38ef730-877a-49cd-98c2-64cbb7894032-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:37.755067 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.755056    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38ef730-877a-49cd-98c2-64cbb7894032-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:37.904467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.904373    5428 generic.go:356] "Generic (PLEG): container finished" podID="f38ef730-877a-49cd-98c2-64cbb7894032" containerID="f5393fcbecf5d6b18b53ec49053e19d9313dbab81a6f69e343572f7c614eb30e" exitCode=0
Oct 29 23:45:37.904467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.904419    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"
Oct 29 23:45:37.904467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.904454    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d" event={"ID":"f38ef730-877a-49cd-98c2-64cbb7894032","Type":"ContainerDied","Data":"f5393fcbecf5d6b18b53ec49053e19d9313dbab81a6f69e343572f7c614eb30e"}
Oct 29 23:45:37.904732 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.904510    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d" event={"ID":"f38ef730-877a-49cd-98c2-64cbb7894032","Type":"ContainerDied","Data":"fd4ab3f364f2e6d6ad1a4e1a1d4c80d9d9812a5e183017315ad333cd06908225"}
Oct 29 23:45:37.904732 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.904544    5428 scope.go:122] "RemoveContainer" containerID="f5393fcbecf5d6b18b53ec49053e19d9313dbab81a6f69e343572f7c614eb30e"
Oct 29 23:45:37.920753 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.920723    5428 scope.go:122] "RemoveContainer" containerID="f5393fcbecf5d6b18b53ec49053e19d9313dbab81a6f69e343572f7c614eb30e"
Oct 29 23:45:37.924220 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:37.924189    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5393fcbecf5d6b18b53ec49053e19d9313dbab81a6f69e343572f7c614eb30e\": container with ID starting with f5393fcbecf5d6b18b53ec49053e19d9313dbab81a6f69e343572f7c614eb30e not found: ID does not exist" containerID="f5393fcbecf5d6b18b53ec49053e19d9313dbab81a6f69e343572f7c614eb30e"
Oct 29 23:45:37.924349 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.924226    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5393fcbecf5d6b18b53ec49053e19d9313dbab81a6f69e343572f7c614eb30e"} err="failed to get container status \"f5393fcbecf5d6b18b53ec49053e19d9313dbab81a6f69e343572f7c614eb30e\": rpc error: code = NotFound desc = could not find container \"f5393fcbecf5d6b18b53ec49053e19d9313dbab81a6f69e343572f7c614eb30e\": container with ID starting with f5393fcbecf5d6b18b53ec49053e19d9313dbab81a6f69e343572f7c614eb30e not found: ID does not exist"
Oct 29 23:45:37.961463 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.961431    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"]
Oct 29 23:45:37.966249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:37.966213    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55c3fc-7448864546-fkk7d"]
Oct 29 23:45:38.388677 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:38.388645    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:38.737758 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:38.737681    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:38.779869 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:38.779841    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f38ef730-877a-49cd-98c2-64cbb7894032" path="/var/lib/kubelet/pods/f38ef730-877a-49cd-98c2-64cbb7894032/volumes"
Oct 29 23:45:38.888182 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:38.888147    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:39.405332 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:39.405298    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:40.550308 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:40.550264    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:40.563462 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:40.563429    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:40.858517 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:40.858440    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:40.954787 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:40.954758    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:45:41.669360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:41.669318    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5" podUID="baf2a48c-3aa1-47fb-aa76-618f73d57163" containerName="payload-pre-processing" containerID="cri-o://c30d61f68348f49e1355a58ff28970aa6e0ccc1ec1b003cd3282f725e3c65f35" gracePeriod=25
Oct 29 23:45:42.670943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:42.670902    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:44.143513 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:44.143482    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:44.825730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:44.825699    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:45.393501 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:45.393453    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5" podUID="baf2a48c-3aa1-47fb-aa76-618f73d57163" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:45:46.779742 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:46.779708    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:47.380938 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:47.380899    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:45:47.510513 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:47.510480    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:49.041889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:49.041846    5428 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","podb288d408-943e-4b45-abc9-7d942e5eb3f2"] err="unable to destroy cgroup paths for cgroup [kubepods burstable podb288d408-943e-4b45-abc9-7d942e5eb3f2] : Timed out while waiting for systemd to remove kubepods-burstable-podb288d408_943e_4b45_abc9_7d942e5eb3f2.slice"
Oct 29 23:45:49.041889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:49.041890    5428 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="b288d408-943e-4b45-abc9-7d942e5eb3f2" err="failed to delete cgroup paths for [kubepods burstable podb288d408-943e-4b45-abc9-7d942e5eb3f2] : unable to destroy cgroup paths for cgroup [kubepods burstable podb288d408-943e-4b45-abc9-7d942e5eb3f2] : Timed out while waiting for systemd to remove kubepods-burstable-podb288d408_943e_4b45_abc9_7d942e5eb3f2.slice"
Oct 29 23:45:49.494649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:49.494616    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:49.615324 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:49.615295    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr"]
Oct 29 23:45:49.738932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:49.738898    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:49.858547 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:49.858461    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:51.899628 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:51.899597    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:52.086465 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:52.086427    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:52.213961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:52.213878    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:52.456325 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:52.456294    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:52.995599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:52.995567    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:53.954628 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:53.954596    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:54.092605 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:54.092570    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:54.616180 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:54.616140    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr" podUID="03ed6510-aaf7-4059-9f80-8cf1595e9c89" containerName="payload-pre-processing" containerID="cri-o://e8df1e85e46f3e2a09e9dc9c2a67f13ed74def5e7e36432f09849007ac2f4319" gracePeriod=25
Oct 29 23:45:54.947463 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:54.947435    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr"
Oct 29 23:45:55.037923 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:55.037889    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ed6510-aaf7-4059-9f80-8cf1595e9c89-kube-api-access-zkqhr\" (UniqueName: \"kubernetes.io/projected/03ed6510-aaf7-4059-9f80-8cf1595e9c89-kube-api-access-zkqhr\") pod \"03ed6510-aaf7-4059-9f80-8cf1595e9c89\" (UID: \"03ed6510-aaf7-4059-9f80-8cf1595e9c89\") "
Oct 29 23:45:55.038102 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:55.038069    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ed6510-aaf7-4059-9f80-8cf1595e9c89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ed6510-aaf7-4059-9f80-8cf1595e9c89-plugins-config-volume\") pod \"03ed6510-aaf7-4059-9f80-8cf1595e9c89\" (UID: \"03ed6510-aaf7-4059-9f80-8cf1595e9c89\") "
Oct 29 23:45:55.038501 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:55.038473    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ed6510-aaf7-4059-9f80-8cf1595e9c89-plugins-config-volume" pod "03ed6510-aaf7-4059-9f80-8cf1595e9c89" (UID: "03ed6510-aaf7-4059-9f80-8cf1595e9c89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:55.039004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:55.038981    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ed6510-aaf7-4059-9f80-8cf1595e9c89-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:55.043246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:55.043200    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ed6510-aaf7-4059-9f80-8cf1595e9c89-kube-api-access-zkqhr" pod "03ed6510-aaf7-4059-9f80-8cf1595e9c89" (UID: "03ed6510-aaf7-4059-9f80-8cf1595e9c89"). InnerVolumeSpecName "kube-api-access-zkqhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:55.067685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:55.067653    5428 generic.go:356] "Generic (PLEG): container finished" podID="03ed6510-aaf7-4059-9f80-8cf1595e9c89" containerID="e8df1e85e46f3e2a09e9dc9c2a67f13ed74def5e7e36432f09849007ac2f4319" exitCode=0
Oct 29 23:45:55.067844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:55.067697    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr"
Oct 29 23:45:55.067844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:55.067729    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr" event={"ID":"03ed6510-aaf7-4059-9f80-8cf1595e9c89","Type":"ContainerDied","Data":"e8df1e85e46f3e2a09e9dc9c2a67f13ed74def5e7e36432f09849007ac2f4319"}
Oct 29 23:45:55.067844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:55.067786    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr" event={"ID":"03ed6510-aaf7-4059-9f80-8cf1595e9c89","Type":"ContainerDied","Data":"75c74ea8b20dbfd44e38c50985cbec0336c175c713cd87b21a93517264a427b5"}
Oct 29 23:45:55.067844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:55.067831    5428 scope.go:122] "RemoveContainer" containerID="e8df1e85e46f3e2a09e9dc9c2a67f13ed74def5e7e36432f09849007ac2f4319"
Oct 29 23:45:55.080736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:55.080719    5428 scope.go:122] "RemoveContainer" containerID="e8df1e85e46f3e2a09e9dc9c2a67f13ed74def5e7e36432f09849007ac2f4319"
Oct 29 23:45:55.081046 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:45:55.081021    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8df1e85e46f3e2a09e9dc9c2a67f13ed74def5e7e36432f09849007ac2f4319\": container with ID starting with e8df1e85e46f3e2a09e9dc9c2a67f13ed74def5e7e36432f09849007ac2f4319 not found: ID does not exist" containerID="e8df1e85e46f3e2a09e9dc9c2a67f13ed74def5e7e36432f09849007ac2f4319"
Oct 29 23:45:55.081131 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:55.081056    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8df1e85e46f3e2a09e9dc9c2a67f13ed74def5e7e36432f09849007ac2f4319"} err="failed to get container status \"e8df1e85e46f3e2a09e9dc9c2a67f13ed74def5e7e36432f09849007ac2f4319\": rpc error: code = NotFound desc = could not find container \"e8df1e85e46f3e2a09e9dc9c2a67f13ed74def5e7e36432f09849007ac2f4319\": container with ID starting with e8df1e85e46f3e2a09e9dc9c2a67f13ed74def5e7e36432f09849007ac2f4319 not found: ID does not exist"
Oct 29 23:45:55.122311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:55.122219    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr"]
Oct 29 23:45:55.129846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:55.129822    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7d4110-6ff565f4cf-5htcr"]
Oct 29 23:45:55.140279 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:55.140256    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkqhr\" (UniqueName: \"kubernetes.io/projected/03ed6510-aaf7-4059-9f80-8cf1595e9c89-kube-api-access-zkqhr\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:45:55.388464 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:55.388430    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5" podUID="baf2a48c-3aa1-47fb-aa76-618f73d57163" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:45:55.667251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:55.667151    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:55.892227 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:55.892188    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:56.780770 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:56.780736    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ed6510-aaf7-4059-9f80-8cf1595e9c89" path="/var/lib/kubelet/pods/03ed6510-aaf7-4059-9f80-8cf1595e9c89/volumes"
Oct 29 23:45:57.882820 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:57.882771    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:57.990856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:57.990823    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"]
Oct 29 23:45:58.003758 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:58.003727    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"]
Oct 29 23:45:58.009113 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:58.009089    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"]
Oct 29 23:45:58.115956 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:58.115923    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:58.605295 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:58.605263    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269"]
Oct 29 23:45:58.637773 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:58.637744    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf"]
Oct 29 23:45:58.764974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:58.764942    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:58.895703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:58.895668    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:45:59.879896 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:45:59.879865    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:01.787438 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:01.787409    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:01.975529 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:01.975496    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5"
Oct 29 23:46:02.055872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:02.055843    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf2a48c-3aa1-47fb-aa76-618f73d57163-kube-api-access-xnbdd\" (UniqueName: \"kubernetes.io/projected/baf2a48c-3aa1-47fb-aa76-618f73d57163-kube-api-access-xnbdd\") pod \"baf2a48c-3aa1-47fb-aa76-618f73d57163\" (UID: \"baf2a48c-3aa1-47fb-aa76-618f73d57163\") "
Oct 29 23:46:02.056058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:02.055907    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baf2a48c-3aa1-47fb-aa76-618f73d57163-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf2a48c-3aa1-47fb-aa76-618f73d57163-plugins-config-volume\") pod \"baf2a48c-3aa1-47fb-aa76-618f73d57163\" (UID: \"baf2a48c-3aa1-47fb-aa76-618f73d57163\") "
Oct 29 23:46:02.056342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:02.056318    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf2a48c-3aa1-47fb-aa76-618f73d57163-plugins-config-volume" pod "baf2a48c-3aa1-47fb-aa76-618f73d57163" (UID: "baf2a48c-3aa1-47fb-aa76-618f73d57163"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:46:02.056596 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:02.056579    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf2a48c-3aa1-47fb-aa76-618f73d57163-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:02.058338 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:02.058308    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf2a48c-3aa1-47fb-aa76-618f73d57163-kube-api-access-xnbdd" pod "baf2a48c-3aa1-47fb-aa76-618f73d57163" (UID: "baf2a48c-3aa1-47fb-aa76-618f73d57163"). InnerVolumeSpecName "kube-api-access-xnbdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:46:02.133916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:02.133883    5428 generic.go:356] "Generic (PLEG): container finished" podID="baf2a48c-3aa1-47fb-aa76-618f73d57163" containerID="c30d61f68348f49e1355a58ff28970aa6e0ccc1ec1b003cd3282f725e3c65f35" exitCode=0
Oct 29 23:46:02.134065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:02.133928    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5"
Oct 29 23:46:02.134065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:02.133932    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5" event={"ID":"baf2a48c-3aa1-47fb-aa76-618f73d57163","Type":"ContainerDied","Data":"c30d61f68348f49e1355a58ff28970aa6e0ccc1ec1b003cd3282f725e3c65f35"}
Oct 29 23:46:02.134065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:02.133984    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5" event={"ID":"baf2a48c-3aa1-47fb-aa76-618f73d57163","Type":"ContainerDied","Data":"0f777f9aea3b8ddea7035a7eda87f7cd3fe055e79ef214456403f76b2bb02078"}
Oct 29 23:46:02.134065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:02.134006    5428 scope.go:122] "RemoveContainer" containerID="c30d61f68348f49e1355a58ff28970aa6e0ccc1ec1b003cd3282f725e3c65f35"
Oct 29 23:46:02.147888 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:02.147871    5428 scope.go:122] "RemoveContainer" containerID="c30d61f68348f49e1355a58ff28970aa6e0ccc1ec1b003cd3282f725e3c65f35"
Oct 29 23:46:02.148156 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:46:02.148132    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30d61f68348f49e1355a58ff28970aa6e0ccc1ec1b003cd3282f725e3c65f35\": container with ID starting with c30d61f68348f49e1355a58ff28970aa6e0ccc1ec1b003cd3282f725e3c65f35 not found: ID does not exist" containerID="c30d61f68348f49e1355a58ff28970aa6e0ccc1ec1b003cd3282f725e3c65f35"
Oct 29 23:46:02.148271 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:02.148157    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30d61f68348f49e1355a58ff28970aa6e0ccc1ec1b003cd3282f725e3c65f35"} err="failed to get container status \"c30d61f68348f49e1355a58ff28970aa6e0ccc1ec1b003cd3282f725e3c65f35\": rpc error: code = NotFound desc = could not find container \"c30d61f68348f49e1355a58ff28970aa6e0ccc1ec1b003cd3282f725e3c65f35\": container with ID starting with c30d61f68348f49e1355a58ff28970aa6e0ccc1ec1b003cd3282f725e3c65f35 not found: ID does not exist"
Oct 29 23:46:02.157660 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:02.157631    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnbdd\" (UniqueName: \"kubernetes.io/projected/baf2a48c-3aa1-47fb-aa76-618f73d57163-kube-api-access-xnbdd\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:02.189744 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:02.189682    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5"]
Oct 29 23:46:02.194985 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:02.194962    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c83b50-55c9df6ccf-v4pm5"]
Oct 29 23:46:02.786640 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:02.786606    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baf2a48c-3aa1-47fb-aa76-618f73d57163" path="/var/lib/kubelet/pods/baf2a48c-3aa1-47fb-aa76-618f73d57163/volumes"
Oct 29 23:46:03.605778 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:03.605739    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269" podUID="6a7e9b55-f400-4b97-ad21-b33e3354c297" containerName="payload-processing" containerID="cri-o://9244cca83282379d084210fe84a7d68a09dfef081d8f427ab8b6510e4c66abf8" gracePeriod=25
Oct 29 23:46:03.638177 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:03.638146    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf" podUID="e78dd7d7-43b5-4e61-96b9-b040ab1185ed" containerName="payload-pre-processing" containerID="cri-o://ad9e5c82f6e1f5851a4ce6f1dec56f95d3620f178b541708649ecee922d5f4a8" gracePeriod=25
Oct 29 23:46:03.699745 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:03.699710    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc"]
Oct 29 23:46:03.795071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:03.795015    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:03.906942 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:03.906901    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:05.577079 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:05.577042    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:05.965443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:05.965407    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf" podUID="e78dd7d7-43b5-4e61-96b9-b040ab1185ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:46:05.965654 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:05.965408    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269" podUID="6a7e9b55-f400-4b97-ad21-b33e3354c297" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:46:07.396815 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:07.396772    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:08.700361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:08.700313    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc" podUID="8b10d05c-ccd8-4125-819e-19e71a233aa9" containerName="payload-pre-processing" containerID="cri-o://92d5122dcc6d92b3449f39354b078177d5e9d1ccf559028ef57df6b6a24844dc" gracePeriod=25
Oct 29 23:46:09.177481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:09.177449    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:10.889985 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:10.889950    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:11.253911 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:11.253780    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc" podUID="8b10d05c-ccd8-4125-819e-19e71a233aa9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:46:12.680692 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:12.680655    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:13.030020 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.029931    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main" containerID="cri-o://dfdf37b1df21d6ddbe2f6e0643e1fdbad5e0dd48af35632c362d4a99932aff34" gracePeriod=45
Oct 29 23:46:13.030255 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.029933    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main" containerID="cri-o://d4e9a5b5b2c7445d269d14c0c32703b28582924df4662ba2be894ddee81a08da" gracePeriod=45
Oct 29 23:46:13.030255 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.030201    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main" containerID="cri-o://fdd204d364f4e8d88e235dd92f474a38c7a879ab279a99af4b9474bf3207c230" gracePeriod=45
Oct 29 23:46:13.243835 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.243250    5428 generic.go:356] "Generic (PLEG): container finished" podID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerID="dfdf37b1df21d6ddbe2f6e0643e1fdbad5e0dd48af35632c362d4a99932aff34" exitCode=0
Oct 29 23:46:13.243835 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.243448    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" event={"ID":"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9","Type":"ContainerDied","Data":"dfdf37b1df21d6ddbe2f6e0643e1fdbad5e0dd48af35632c362d4a99932aff34"}
Oct 29 23:46:13.246394 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.245867    5428 generic.go:356] "Generic (PLEG): container finished" podID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerID="d4e9a5b5b2c7445d269d14c0c32703b28582924df4662ba2be894ddee81a08da" exitCode=0
Oct 29 23:46:13.246542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.245965    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" event={"ID":"22d79ec5-027d-4dc6-b4ac-ec7e018c039a","Type":"ContainerDied","Data":"d4e9a5b5b2c7445d269d14c0c32703b28582924df4662ba2be894ddee81a08da"}
Oct 29 23:46:13.247714 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.247693    5428 generic.go:356] "Generic (PLEG): container finished" podID="8df2f41d-d961-4041-9125-685f4d3a375f" containerID="fdd204d364f4e8d88e235dd92f474a38c7a879ab279a99af4b9474bf3207c230" exitCode=0
Oct 29 23:46:13.247918 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.247891    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" event={"ID":"8df2f41d-d961-4041-9125-685f4d3a375f","Type":"ContainerDied","Data":"fdd204d364f4e8d88e235dd92f474a38c7a879ab279a99af4b9474bf3207c230"}
Oct 29 23:46:13.349365 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.349327    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 23:46:13.395210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.395183    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 23:46:13.398163 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.398135    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 23:46:13.440655 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.440592    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8df2f41d-d961-4041-9125-685f4d3a375f-tls-certs\" (UniqueName: \"kubernetes.io/secret/8df2f41d-d961-4041-9125-685f4d3a375f-tls-certs\") pod \"8df2f41d-d961-4041-9125-685f4d3a375f\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") "
Oct 29 23:46:13.440923 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.440728    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-dshm\") pod \"8df2f41d-d961-4041-9125-685f4d3a375f\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") "
Oct 29 23:46:13.440923 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.440848    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df2f41d-d961-4041-9125-685f4d3a375f-kube-api-access-wv72h\" (UniqueName: \"kubernetes.io/projected/8df2f41d-d961-4041-9125-685f4d3a375f-kube-api-access-wv72h\") pod \"8df2f41d-d961-4041-9125-685f4d3a375f\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") "
Oct 29 23:46:13.441148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.440992    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-home\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-home\") pod \"8df2f41d-d961-4041-9125-685f4d3a375f\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") "
Oct 29 23:46:13.441148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.441031    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-tmp-dir\") pod \"8df2f41d-d961-4041-9125-685f4d3a375f\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") "
Oct 29 23:46:13.441148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.441093    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-model-cache\") pod \"8df2f41d-d961-4041-9125-685f4d3a375f\" (UID: \"8df2f41d-d961-4041-9125-685f4d3a375f\") "
Oct 29 23:46:13.442239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.442186    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-model-cache" pod "8df2f41d-d961-4041-9125-685f4d3a375f" (UID: "8df2f41d-d961-4041-9125-685f4d3a375f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:13.466072 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.466019    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-home" pod "8df2f41d-d961-4041-9125-685f4d3a375f" (UID: "8df2f41d-d961-4041-9125-685f4d3a375f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:13.466072 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.466058    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-dshm" pod "8df2f41d-d961-4041-9125-685f4d3a375f" (UID: "8df2f41d-d961-4041-9125-685f4d3a375f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:13.466339 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.466108    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df2f41d-d961-4041-9125-685f4d3a375f-kube-api-access-wv72h" pod "8df2f41d-d961-4041-9125-685f4d3a375f" (UID: "8df2f41d-d961-4041-9125-685f4d3a375f"). InnerVolumeSpecName "kube-api-access-wv72h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:46:13.466339 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.466212    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df2f41d-d961-4041-9125-685f4d3a375f-tls-certs" pod "8df2f41d-d961-4041-9125-685f4d3a375f" (UID: "8df2f41d-d961-4041-9125-685f4d3a375f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:46:13.466737 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.466708    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-tmp-dir" pod "8df2f41d-d961-4041-9125-685f4d3a375f" (UID: "8df2f41d-d961-4041-9125-685f4d3a375f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:13.542814 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.542714    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-model-cache\") pod \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") "
Oct 29 23:46:13.542990 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.542831    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-tls-certs\" (UniqueName: \"kubernetes.io/secret/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-tls-certs\") pod \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") "
Oct 29 23:46:13.542990 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.542877    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-home\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-home\") pod \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") "
Oct 29 23:46:13.542990 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.542947    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-model-cache\") pod \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") "
Oct 29 23:46:13.543167 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.543054    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-dshm\") pod \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") "
Oct 29 23:46:13.543167 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.543106    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-tmp-dir\") pod \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") "
Oct 29 23:46:13.543273 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.543204    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-tls-certs\") pod \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") "
Oct 29 23:46:13.543360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.543281    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-kube-api-access-mrfgx\" (UniqueName: \"kubernetes.io/projected/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-kube-api-access-mrfgx\") pod \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") "
Oct 29 23:46:13.543360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.543315    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-home\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-home\") pod \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") "
Oct 29 23:46:13.543469 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.543364    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-kube-api-access-pqbjc\" (UniqueName: \"kubernetes.io/projected/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-kube-api-access-pqbjc\") pod \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") "
Oct 29 23:46:13.543469 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.543437    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-dshm\") pod \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\" (UID: \"22d79ec5-027d-4dc6-b4ac-ec7e018c039a\") "
Oct 29 23:46:13.543578 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.543497    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-tmp-dir\") pod \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\" (UID: \"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9\") "
Oct 29 23:46:13.543920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.543892    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-model-cache" pod "22d79ec5-027d-4dc6-b4ac-ec7e018c039a" (UID: "22d79ec5-027d-4dc6-b4ac-ec7e018c039a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:13.544287 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.544264    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-home" pod "22d79ec5-027d-4dc6-b4ac-ec7e018c039a" (UID: "22d79ec5-027d-4dc6-b4ac-ec7e018c039a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:13.544425 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.544409    5428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-home\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:13.544526 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.544513    5428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-tmp-dir\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:13.544628 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.544617    5428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-model-cache\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:13.544719 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.544708    5428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-model-cache\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:13.544850 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.544834    5428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df2f41d-d961-4041-9125-685f4d3a375f-tls-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:13.544958 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.544946    5428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df2f41d-d961-4041-9125-685f4d3a375f-dshm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:13.545054 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.545040    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv72h\" (UniqueName: \"kubernetes.io/projected/8df2f41d-d961-4041-9125-685f4d3a375f-kube-api-access-wv72h\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:13.545161 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.544717    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-model-cache" pod "b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" (UID: "b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:13.545161 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.545035    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-tmp-dir" pod "b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" (UID: "b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:13.545355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.545214    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-tmp-dir" pod "22d79ec5-027d-4dc6-b4ac-ec7e018c039a" (UID: "22d79ec5-027d-4dc6-b4ac-ec7e018c039a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:13.545431 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.545391    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-home" pod "b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" (UID: "b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:13.548214 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.548173    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-dshm" pod "b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" (UID: "b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:13.548214 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.548182    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-tls-certs" pod "22d79ec5-027d-4dc6-b4ac-ec7e018c039a" (UID: "22d79ec5-027d-4dc6-b4ac-ec7e018c039a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:46:13.548214 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.548188    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-tls-certs" pod "b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" (UID: "b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:46:13.548441 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.548217    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-kube-api-access-pqbjc" pod "22d79ec5-027d-4dc6-b4ac-ec7e018c039a" (UID: "22d79ec5-027d-4dc6-b4ac-ec7e018c039a"). InnerVolumeSpecName "kube-api-access-pqbjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:46:13.548441 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.548234    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-dshm" pod "22d79ec5-027d-4dc6-b4ac-ec7e018c039a" (UID: "22d79ec5-027d-4dc6-b4ac-ec7e018c039a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:13.548441 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.548383    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-kube-api-access-mrfgx" pod "b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" (UID: "b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9"). InnerVolumeSpecName "kube-api-access-mrfgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:46:13.646192 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.646160    5428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-dshm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:13.646192 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.646187    5428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-tmp-dir\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:13.646192 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.646199    5428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-tls-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:13.646443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.646208    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrfgx\" (UniqueName: \"kubernetes.io/projected/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-kube-api-access-mrfgx\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:13.646443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.646217    5428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-home\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:13.646443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.646225    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqbjc\" (UniqueName: \"kubernetes.io/projected/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-kube-api-access-pqbjc\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:13.646443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.646240    5428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-dshm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:13.646443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.646248    5428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-tmp-dir\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:13.646443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.646257    5428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-tls-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:13.646443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.646264    5428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22d79ec5-027d-4dc6-b4ac-ec7e018c039a-home\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:13.646443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.646272    5428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9-model-cache\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:13.928435 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.928404    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"]
Oct 29 23:46:13.929264 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929248    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9ffbf666-ed06-4d62-bf9a-d1786bcf071f" containerName="cleanup"
Oct 29 23:46:13.929323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929265    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ffbf666-ed06-4d62-bf9a-d1786bcf071f" containerName="cleanup"
Oct 29 23:46:13.929323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929277    5428 cpu_manager.go:414] "containerMap: removing container" podUID="f38ef730-877a-49cd-98c2-64cbb7894032" containerName="maas-api"
Oct 29 23:46:13.929323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929282    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f38ef730-877a-49cd-98c2-64cbb7894032" containerName="maas-api"
Oct 29 23:46:13.929323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929295    5428 cpu_manager.go:414] "containerMap: removing container" podUID="36d961cc-02de-4c2d-8f84-87e66cf01259" containerName="cleanup"
Oct 29 23:46:13.929323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929300    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d961cc-02de-4c2d-8f84-87e66cf01259" containerName="cleanup"
Oct 29 23:46:13.929323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929311    5428 cpu_manager.go:414] "containerMap: removing container" podUID="03ed6510-aaf7-4059-9f80-8cf1595e9c89" containerName="payload-pre-processing"
Oct 29 23:46:13.929323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929316    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ed6510-aaf7-4059-9f80-8cf1595e9c89" containerName="payload-pre-processing"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929337    5428 cpu_manager.go:414] "containerMap: removing container" podUID="baf2a48c-3aa1-47fb-aa76-618f73d57163" containerName="payload-pre-processing"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929342    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf2a48c-3aa1-47fb-aa76-618f73d57163" containerName="payload-pre-processing"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929353    5428 cpu_manager.go:414] "containerMap: removing container" podUID="918c3198-f2b2-413c-9002-91b324b0aeae" containerName="cleanup"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929357    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918c3198-f2b2-413c-9002-91b324b0aeae" containerName="cleanup"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929364    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b0a1fa0e-f264-46bb-83c0-efd1be480dd3" containerName="cleanup"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929369    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a1fa0e-f264-46bb-83c0-efd1be480dd3" containerName="cleanup"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929377    5428 cpu_manager.go:414] "containerMap: removing container" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929381    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929388    5428 cpu_manager.go:414] "containerMap: removing container" podUID="d8fcfb61-2011-477e-9138-c7eb186665a6" containerName="cleanup"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929392    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8fcfb61-2011-477e-9138-c7eb186665a6" containerName="cleanup"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929406    5428 cpu_manager.go:414] "containerMap: removing container" podUID="3ba2006e-ff86-47a3-8d81-c935530476e8" containerName="cleanup"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929410    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ba2006e-ff86-47a3-8d81-c935530476e8" containerName="cleanup"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929420    5428 cpu_manager.go:414] "containerMap: removing container" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929425    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929433    5428 cpu_manager.go:414] "containerMap: removing container" podUID="4ff49c0b-5f9d-411a-a767-77897583a7a3" containerName="cleanup"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929438    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ff49c0b-5f9d-411a-a767-77897583a7a3" containerName="cleanup"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929452    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929456    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929557    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ed6510-aaf7-4059-9f80-8cf1595e9c89" containerName="payload-pre-processing"
Oct 29 23:46:13.929561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929568    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ba2006e-ff86-47a3-8d81-c935530476e8" containerName="cleanup"
Oct 29 23:46:13.930208 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929574    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ff49c0b-5f9d-411a-a767-77897583a7a3" containerName="cleanup"
Oct 29 23:46:13.930208 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929582    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf2a48c-3aa1-47fb-aa76-618f73d57163" containerName="payload-pre-processing"
Oct 29 23:46:13.930208 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929588    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" containerName="main"
Oct 29 23:46:13.930208 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929597    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" containerName="main"
Oct 29 23:46:13.930208 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929604    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" containerName="main"
Oct 29 23:46:13.930208 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929610    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc6e9365-f21c-49b3-b513-c17c1113cb41" containerName="cleanup"
Oct 29 23:46:13.930208 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929617    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f38ef730-877a-49cd-98c2-64cbb7894032" containerName="maas-api"
Oct 29 23:46:13.930208 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.929625    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d961cc-02de-4c2d-8f84-87e66cf01259" containerName="cleanup"
Oct 29 23:46:13.935121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.935099    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:13.961987 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:13.960993    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"]
Oct 29 23:46:14.054950 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.054920    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.055071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.054978    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.055142 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.055104    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp48x\" (UniqueName: \"kubernetes.io/projected/43dcb139-963e-4266-852b-d0119b2d7b0e-kube-api-access-bp48x\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.055207 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.055167    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.055269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.055211    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.055269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.055237    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.055370 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.055344    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.055480 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.055462    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.055532 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.055520    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43dcb139-963e-4266-852b-d0119b2d7b0e-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.157406 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.157376    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.157572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.157437    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp48x\" (UniqueName: \"kubernetes.io/projected/43dcb139-963e-4266-852b-d0119b2d7b0e-kube-api-access-bp48x\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.157572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.157485    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.157572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.157517    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.157572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.157532    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.157572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.157561    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.157916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.157641    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.157916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.157706    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43dcb139-963e-4266-852b-d0119b2d7b0e-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.157916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.157906    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.158091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.158017    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.158091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.158057    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.158091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.158074    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.158664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.158286    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.158664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.158655    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43dcb139-963e-4266-852b-d0119b2d7b0e-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.159618 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.159595    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.159998 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.159978    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.180496 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.180436    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.180921 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.180901    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp48x\" (UniqueName: \"kubernetes.io/projected/43dcb139-963e-4266-852b-d0119b2d7b0e-kube-api-access-bp48x\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.257888 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.257847    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h" event={"ID":"b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9","Type":"ContainerDied","Data":"92d87a1b7449c68ed2ae7a4b3662b3223b9277383ea4f906f93da18bff6fa8b7"}
Oct 29 23:46:14.258093 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.257897    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"
Oct 29 23:46:14.258093 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.257931    5428 scope.go:122] "RemoveContainer" containerID="dfdf37b1df21d6ddbe2f6e0643e1fdbad5e0dd48af35632c362d4a99932aff34"
Oct 29 23:46:14.258831 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.258784    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:14.268762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.262790    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"
Oct 29 23:46:14.268762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.263174    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft" event={"ID":"22d79ec5-027d-4dc6-b4ac-ec7e018c039a","Type":"ContainerDied","Data":"0501c3f90a31f6bbd1e7a3afd1805bde45a3b46291ce0f75a256daa44ab98201"}
Oct 29 23:46:14.268762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.264543    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"
Oct 29 23:46:14.268762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.264606    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6" event={"ID":"8df2f41d-d961-4041-9125-685f4d3a375f","Type":"ContainerDied","Data":"c79d92f093061d63a23ec65e46f282e5f48d2a3c5911a7cfaf0b9607ae35ba0e"}
Oct 29 23:46:14.282531 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.282503    5428 scope.go:122] "RemoveContainer" containerID="d4e9a5b5b2c7445d269d14c0c32703b28582924df4662ba2be894ddee81a08da"
Oct 29 23:46:14.297191 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.297173    5428 scope.go:122] "RemoveContainer" containerID="fdd204d364f4e8d88e235dd92f474a38c7a879ab279a99af4b9474bf3207c230"
Oct 29 23:46:14.344917 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.344873    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"]
Oct 29 23:46:14.353435 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.353400    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/distinct-w2-7aeaad-kserve-7f56b5df96-x7d7h"]
Oct 29 23:46:14.385044 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.385014    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"]
Oct 29 23:46:14.392813 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.392772    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/premium-w2-7aeaad-kserve-7864b989bf-bpzq6"]
Oct 29 23:46:14.418298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.418261    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"]
Oct 29 23:46:14.422790 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.422760    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7aeaad/distinct-2-w2-7aeaad-kserve-647985bff-67bft"]
Oct 29 23:46:14.607932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.607902    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"]
Oct 29 23:46:14.608192 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:46:14.608170    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43dcb139_963e_4266_852b_d0119b2d7b0e.slice/crio-c19121fcc8df3e92bfcbee5a9a18a9c807a8db3fab0126738c0b821f1f68f5cb WatchSource:0}: Error finding container c19121fcc8df3e92bfcbee5a9a18a9c807a8db3fab0126738c0b821f1f68f5cb: Status 404 returned error can't find the container with id c19121fcc8df3e92bfcbee5a9a18a9c807a8db3fab0126738c0b821f1f68f5cb
Oct 29 23:46:14.611316 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.611282    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:46:14.611448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.611360    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:46:14.611448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.611405    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:46:14.688732 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.688709    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:14.784937 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.783382    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22d79ec5-027d-4dc6-b4ac-ec7e018c039a" path="/var/lib/kubelet/pods/22d79ec5-027d-4dc6-b4ac-ec7e018c039a/volumes"
Oct 29 23:46:14.784937 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.784051    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8df2f41d-d961-4041-9125-685f4d3a375f" path="/var/lib/kubelet/pods/8df2f41d-d961-4041-9125-685f4d3a375f/volumes"
Oct 29 23:46:14.785406 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:14.785380    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9" path="/var/lib/kubelet/pods/b5ba8e0a-e30f-4663-bb3f-1fab6ec430e9/volumes"
Oct 29 23:46:15.277015 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:15.276978    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h" event={"ID":"43dcb139-963e-4266-852b-d0119b2d7b0e","Type":"ContainerStarted","Data":"db1e30c2e05fbf80c359e6d1e6037e6a0345f7450dff31bf8c79bcb3f424d66b"}
Oct 29 23:46:15.277486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:15.277037    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h" event={"ID":"43dcb139-963e-4266-852b-d0119b2d7b0e","Type":"ContainerStarted","Data":"c19121fcc8df3e92bfcbee5a9a18a9c807a8db3fab0126738c0b821f1f68f5cb"}
Oct 29 23:46:15.327348 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:15.327291    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h" podStartSLOduration=2.327273355 podStartE2EDuration="2.327273355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:46:15.32070496 +0000 UTC m=+3778.663653739" watchObservedRunningTime="2026-10-29 23:46:15.327273355 +0000 UTC m=+3778.670222104"
Oct 29 23:46:15.985048 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:15.984994    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf" podUID="e78dd7d7-43b5-4e61-96b9-b040ab1185ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:46:15.986217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:15.986180    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269" podUID="6a7e9b55-f400-4b97-ad21-b33e3354c297" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:46:16.287017 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:16.283553    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:16.289249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:16.289220    5428 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Oct 29 23:46:16.289478 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:16.289453    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h" podUID="43dcb139-963e-4266-852b-d0119b2d7b0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Oct 29 23:46:16.714423 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:16.714386    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:17.277826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:17.266149    5428 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Oct 29 23:46:17.277826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:17.266212    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h" podUID="43dcb139-963e-4266-852b-d0119b2d7b0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Oct 29 23:46:17.392926 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:46:17.392884    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:46:18.270825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:18.267135    5428 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Oct 29 23:46:18.270825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:18.267194    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h" podUID="43dcb139-963e-4266-852b-d0119b2d7b0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Oct 29 23:46:18.713413 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:18.713384    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:19.284013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:19.283929    5428 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Oct 29 23:46:19.284212 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:19.284016    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h" podUID="43dcb139-963e-4266-852b-d0119b2d7b0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Oct 29 23:46:20.265300 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:20.265265    5428 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Oct 29 23:46:20.265873 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:20.265841    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h" podUID="43dcb139-963e-4266-852b-d0119b2d7b0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Oct 29 23:46:20.519410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:20.519327    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:21.253246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:21.253205    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc" podUID="8b10d05c-ccd8-4125-819e-19e71a233aa9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:46:21.265134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:21.265105    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:21.265607 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:21.265588    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:21.266275 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:21.266254    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:46:22.282140 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:22.282107    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:22.962864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:22.962823    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"]
Oct 29 23:46:22.969563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:22.969536    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:22.983585 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:22.983550    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"]
Oct 29 23:46:22.988379 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:22.988352    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-adc7d7\"/\"test-model-adc7d7-kserve-self-signed-certs\""
Oct 29 23:46:22.988606 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:22.988469    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-adc7d7\"/\"default-dockercfg-72p5f\""
Oct 29 23:46:23.000267 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.000247    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-adc7d7\"/\"kube-root-ca.crt\""
Oct 29 23:46:23.000267 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.000254    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-adc7d7\"/\"openshift-service-ca.crt\""
Oct 29 23:46:23.006752 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.006728    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:23.126900 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.126862    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c84aca64-8103-4122-9899-158745522b85-tls-certs\") pod \"test-model-adc7d7-kserve-8fdcd9d79-g2t29\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") " pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.127070 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.127026    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-dshm\") pod \"test-model-adc7d7-kserve-8fdcd9d79-g2t29\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") " pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.127151 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.127084    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-tmp-dir\") pod \"test-model-adc7d7-kserve-8fdcd9d79-g2t29\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") " pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.127261 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.127239    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcfg9\" (UniqueName: \"kubernetes.io/projected/c84aca64-8103-4122-9899-158745522b85-kube-api-access-zcfg9\") pod \"test-model-adc7d7-kserve-8fdcd9d79-g2t29\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") " pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.127327 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.127289    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-home\") pod \"test-model-adc7d7-kserve-8fdcd9d79-g2t29\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") " pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.127381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.127364    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-model-cache\") pod \"test-model-adc7d7-kserve-8fdcd9d79-g2t29\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") " pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.229786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.229685    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c84aca64-8103-4122-9899-158745522b85-tls-certs\") pod \"test-model-adc7d7-kserve-8fdcd9d79-g2t29\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") " pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.229999 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.229881    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-dshm\") pod \"test-model-adc7d7-kserve-8fdcd9d79-g2t29\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") " pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.229999 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.229938    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-tmp-dir\") pod \"test-model-adc7d7-kserve-8fdcd9d79-g2t29\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") " pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.229999 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.229998    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcfg9\" (UniqueName: \"kubernetes.io/projected/c84aca64-8103-4122-9899-158745522b85-kube-api-access-zcfg9\") pod \"test-model-adc7d7-kserve-8fdcd9d79-g2t29\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") " pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.230296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.230263    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-home\") pod \"test-model-adc7d7-kserve-8fdcd9d79-g2t29\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") " pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.230522 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.230497    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-model-cache\") pod \"test-model-adc7d7-kserve-8fdcd9d79-g2t29\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") " pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.230645 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.230503    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-home\") pod \"test-model-adc7d7-kserve-8fdcd9d79-g2t29\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") " pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.230717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.230380    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-tmp-dir\") pod \"test-model-adc7d7-kserve-8fdcd9d79-g2t29\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") " pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.230717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.230678    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-model-cache\") pod \"test-model-adc7d7-kserve-8fdcd9d79-g2t29\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") " pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.232642 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.232620    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c84aca64-8103-4122-9899-158745522b85-tls-certs\") pod \"test-model-adc7d7-kserve-8fdcd9d79-g2t29\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") " pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.232738 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.232676    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-dshm\") pod \"test-model-adc7d7-kserve-8fdcd9d79-g2t29\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") " pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.262006 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.261974    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcfg9\" (UniqueName: \"kubernetes.io/projected/c84aca64-8103-4122-9899-158745522b85-kube-api-access-zcfg9\") pod \"test-model-adc7d7-kserve-8fdcd9d79-g2t29\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") " pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.298188 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.298159    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:23.612394 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.612359    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"]
Oct 29 23:46:23.939700 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.939655    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269"
Oct 29 23:46:23.977280 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:23.977247    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf"
Oct 29 23:46:24.046825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.046092    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7e9b55-f400-4b97-ad21-b33e3354c297-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7e9b55-f400-4b97-ad21-b33e3354c297-plugins-config-volume\") pod \"6a7e9b55-f400-4b97-ad21-b33e3354c297\" (UID: \"6a7e9b55-f400-4b97-ad21-b33e3354c297\") "
Oct 29 23:46:24.046825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.046311    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7e9b55-f400-4b97-ad21-b33e3354c297-kube-api-access-wnjg2\" (UniqueName: \"kubernetes.io/projected/6a7e9b55-f400-4b97-ad21-b33e3354c297-kube-api-access-wnjg2\") pod \"6a7e9b55-f400-4b97-ad21-b33e3354c297\" (UID: \"6a7e9b55-f400-4b97-ad21-b33e3354c297\") "
Oct 29 23:46:24.046825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.046527    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7e9b55-f400-4b97-ad21-b33e3354c297-plugins-config-volume" pod "6a7e9b55-f400-4b97-ad21-b33e3354c297" (UID: "6a7e9b55-f400-4b97-ad21-b33e3354c297"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:46:24.048846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.047127    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7e9b55-f400-4b97-ad21-b33e3354c297-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:24.049534 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.049506    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7e9b55-f400-4b97-ad21-b33e3354c297-kube-api-access-wnjg2" pod "6a7e9b55-f400-4b97-ad21-b33e3354c297" (UID: "6a7e9b55-f400-4b97-ad21-b33e3354c297"). InnerVolumeSpecName "kube-api-access-wnjg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:46:24.151174 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.151141    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78dd7d7-43b5-4e61-96b9-b040ab1185ed-kube-api-access-77gtl\" (UniqueName: \"kubernetes.io/projected/e78dd7d7-43b5-4e61-96b9-b040ab1185ed-kube-api-access-77gtl\") pod \"e78dd7d7-43b5-4e61-96b9-b040ab1185ed\" (UID: \"e78dd7d7-43b5-4e61-96b9-b040ab1185ed\") "
Oct 29 23:46:24.151499 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.151477    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e78dd7d7-43b5-4e61-96b9-b040ab1185ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78dd7d7-43b5-4e61-96b9-b040ab1185ed-plugins-config-volume\") pod \"e78dd7d7-43b5-4e61-96b9-b040ab1185ed\" (UID: \"e78dd7d7-43b5-4e61-96b9-b040ab1185ed\") "
Oct 29 23:46:24.151912 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.151883    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78dd7d7-43b5-4e61-96b9-b040ab1185ed-plugins-config-volume" pod "e78dd7d7-43b5-4e61-96b9-b040ab1185ed" (UID: "e78dd7d7-43b5-4e61-96b9-b040ab1185ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:46:24.152108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.152087    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnjg2\" (UniqueName: \"kubernetes.io/projected/6a7e9b55-f400-4b97-ad21-b33e3354c297-kube-api-access-wnjg2\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:24.152108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.152107    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78dd7d7-43b5-4e61-96b9-b040ab1185ed-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:24.153364 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.153335    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78dd7d7-43b5-4e61-96b9-b040ab1185ed-kube-api-access-77gtl" pod "e78dd7d7-43b5-4e61-96b9-b040ab1185ed" (UID: "e78dd7d7-43b5-4e61-96b9-b040ab1185ed"). InnerVolumeSpecName "kube-api-access-77gtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:46:24.254064 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.254032    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77gtl\" (UniqueName: \"kubernetes.io/projected/e78dd7d7-43b5-4e61-96b9-b040ab1185ed-kube-api-access-77gtl\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:24.378321 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.378278    5428 generic.go:356] "Generic (PLEG): container finished" podID="6a7e9b55-f400-4b97-ad21-b33e3354c297" containerID="9244cca83282379d084210fe84a7d68a09dfef081d8f427ab8b6510e4c66abf8" exitCode=0
Oct 29 23:46:24.378894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.378340    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269"
Oct 29 23:46:24.378894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.378372    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269" event={"ID":"6a7e9b55-f400-4b97-ad21-b33e3354c297","Type":"ContainerDied","Data":"9244cca83282379d084210fe84a7d68a09dfef081d8f427ab8b6510e4c66abf8"}
Oct 29 23:46:24.378894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.378446    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269" event={"ID":"6a7e9b55-f400-4b97-ad21-b33e3354c297","Type":"ContainerDied","Data":"8ee5ec03dd0f527679f970bf12b6c61d75d08cc82998b09a5b0418af3b88282f"}
Oct 29 23:46:24.378894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.378484    5428 scope.go:122] "RemoveContainer" containerID="9244cca83282379d084210fe84a7d68a09dfef081d8f427ab8b6510e4c66abf8"
Oct 29 23:46:24.382379 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.382348    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29" event={"ID":"c84aca64-8103-4122-9899-158745522b85","Type":"ContainerStarted","Data":"cd62f1ab5c28697edde0eaf3074d837b9ea37123c31f220fefbff796954c0789"}
Oct 29 23:46:24.382507 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.382391    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29" event={"ID":"c84aca64-8103-4122-9899-158745522b85","Type":"ContainerStarted","Data":"d2816bbfa89bebd31d3d1d7528484abb5caf34ee998ea0935cb8a12ba70796cb"}
Oct 29 23:46:24.385359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.385076    5428 generic.go:356] "Generic (PLEG): container finished" podID="e78dd7d7-43b5-4e61-96b9-b040ab1185ed" containerID="ad9e5c82f6e1f5851a4ce6f1dec56f95d3620f178b541708649ecee922d5f4a8" exitCode=0
Oct 29 23:46:24.385359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.385183    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf" event={"ID":"e78dd7d7-43b5-4e61-96b9-b040ab1185ed","Type":"ContainerDied","Data":"ad9e5c82f6e1f5851a4ce6f1dec56f95d3620f178b541708649ecee922d5f4a8"}
Oct 29 23:46:24.385359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.385225    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf" event={"ID":"e78dd7d7-43b5-4e61-96b9-b040ab1185ed","Type":"ContainerDied","Data":"d6c4c8f93bec29c08d17f5d12ac6155c97995df0777ad3b8a048415efae9ed8e"}
Oct 29 23:46:24.385359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.385291    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf"
Oct 29 23:46:24.395450 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.395427    5428 scope.go:122] "RemoveContainer" containerID="9244cca83282379d084210fe84a7d68a09dfef081d8f427ab8b6510e4c66abf8"
Oct 29 23:46:24.395789 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:46:24.395760    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9244cca83282379d084210fe84a7d68a09dfef081d8f427ab8b6510e4c66abf8\": container with ID starting with 9244cca83282379d084210fe84a7d68a09dfef081d8f427ab8b6510e4c66abf8 not found: ID does not exist" containerID="9244cca83282379d084210fe84a7d68a09dfef081d8f427ab8b6510e4c66abf8"
Oct 29 23:46:24.395789 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.395796    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9244cca83282379d084210fe84a7d68a09dfef081d8f427ab8b6510e4c66abf8"} err="failed to get container status \"9244cca83282379d084210fe84a7d68a09dfef081d8f427ab8b6510e4c66abf8\": rpc error: code = NotFound desc = could not find container \"9244cca83282379d084210fe84a7d68a09dfef081d8f427ab8b6510e4c66abf8\": container with ID starting with 9244cca83282379d084210fe84a7d68a09dfef081d8f427ab8b6510e4c66abf8 not found: ID does not exist"
Oct 29 23:46:24.395789 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.395832    5428 scope.go:122] "RemoveContainer" containerID="ad9e5c82f6e1f5851a4ce6f1dec56f95d3620f178b541708649ecee922d5f4a8"
Oct 29 23:46:24.396445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.396417    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:24.409908 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.409882    5428 scope.go:122] "RemoveContainer" containerID="ad9e5c82f6e1f5851a4ce6f1dec56f95d3620f178b541708649ecee922d5f4a8"
Oct 29 23:46:24.410278 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:46:24.410254    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad9e5c82f6e1f5851a4ce6f1dec56f95d3620f178b541708649ecee922d5f4a8\": container with ID starting with ad9e5c82f6e1f5851a4ce6f1dec56f95d3620f178b541708649ecee922d5f4a8 not found: ID does not exist" containerID="ad9e5c82f6e1f5851a4ce6f1dec56f95d3620f178b541708649ecee922d5f4a8"
Oct 29 23:46:24.410352 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.410287    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad9e5c82f6e1f5851a4ce6f1dec56f95d3620f178b541708649ecee922d5f4a8"} err="failed to get container status \"ad9e5c82f6e1f5851a4ce6f1dec56f95d3620f178b541708649ecee922d5f4a8\": rpc error: code = NotFound desc = could not find container \"ad9e5c82f6e1f5851a4ce6f1dec56f95d3620f178b541708649ecee922d5f4a8\": container with ID starting with ad9e5c82f6e1f5851a4ce6f1dec56f95d3620f178b541708649ecee922d5f4a8 not found: ID does not exist"
Oct 29 23:46:24.439329 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.439244    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29" podStartSLOduration=2.439222884 podStartE2EDuration="2.439222884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:46:24.436594516 +0000 UTC m=+3787.779543261" watchObservedRunningTime="2026-10-29 23:46:24.439222884 +0000 UTC m=+3787.782171630"
Oct 29 23:46:24.498186 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.498148    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf"]
Oct 29 23:46:24.505045 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.504963    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6f162a-6857685b99-9s8tf"]
Oct 29 23:46:24.512281 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.512254    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:24.529525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.529493    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269"]
Oct 29 23:46:24.535627 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.535605    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6f162a-785f89fcb8-dv269"]
Oct 29 23:46:24.705429 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.705399    5428 scope.go:122] "RemoveContainer" containerID="73fb9dd75b3db4a49e404688c01cca8f4b3dc20f427f107dd8c07c2b674c6806"
Oct 29 23:46:24.785258 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.785229    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a7e9b55-f400-4b97-ad21-b33e3354c297" path="/var/lib/kubelet/pods/6a7e9b55-f400-4b97-ad21-b33e3354c297/volumes"
Oct 29 23:46:24.785898 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:24.785877    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e78dd7d7-43b5-4e61-96b9-b040ab1185ed" path="/var/lib/kubelet/pods/e78dd7d7-43b5-4e61-96b9-b040ab1185ed/volumes"
Oct 29 23:46:25.302927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:25.302893    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:26.374509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:26.374454    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:27.318026 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:27.317982    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:27.589955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:27.589863    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:27.748645 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:27.748435    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:28.190511 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:28.190474    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:28.962667 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:28.962565    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"]
Oct 29 23:46:28.963888 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:28.963825    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e78dd7d7-43b5-4e61-96b9-b040ab1185ed" containerName="payload-pre-processing"
Oct 29 23:46:28.963888 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:28.963847    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78dd7d7-43b5-4e61-96b9-b040ab1185ed" containerName="payload-pre-processing"
Oct 29 23:46:28.964397 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:28.963898    5428 cpu_manager.go:414] "containerMap: removing container" podUID="6a7e9b55-f400-4b97-ad21-b33e3354c297" containerName="payload-processing"
Oct 29 23:46:28.964397 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:28.963908    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7e9b55-f400-4b97-ad21-b33e3354c297" containerName="payload-processing"
Oct 29 23:46:28.964397 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:28.964053    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7e9b55-f400-4b97-ad21-b33e3354c297" containerName="payload-processing"
Oct 29 23:46:28.964397 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:28.964074    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78dd7d7-43b5-4e61-96b9-b040ab1185ed" containerName="payload-pre-processing"
Oct 29 23:46:28.967568 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:28.967545    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:28.983614 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:28.983586    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"]
Oct 29 23:46:29.036914 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.036885    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc"
Oct 29 23:46:29.131495 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.131466    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b10d05c-ccd8-4125-819e-19e71a233aa9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b10d05c-ccd8-4125-819e-19e71a233aa9-plugins-config-volume\") pod \"8b10d05c-ccd8-4125-819e-19e71a233aa9\" (UID: \"8b10d05c-ccd8-4125-819e-19e71a233aa9\") "
Oct 29 23:46:29.131730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.131712    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b10d05c-ccd8-4125-819e-19e71a233aa9-kube-api-access-p2rjm\" (UniqueName: \"kubernetes.io/projected/8b10d05c-ccd8-4125-819e-19e71a233aa9-kube-api-access-p2rjm\") pod \"8b10d05c-ccd8-4125-819e-19e71a233aa9\" (UID: \"8b10d05c-ccd8-4125-819e-19e71a233aa9\") "
Oct 29 23:46:29.132087 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.132036    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b10d05c-ccd8-4125-819e-19e71a233aa9-plugins-config-volume" pod "8b10d05c-ccd8-4125-819e-19e71a233aa9" (UID: "8b10d05c-ccd8-4125-819e-19e71a233aa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:46:29.132200 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.132173    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p74q7\" (UniqueName: \"kubernetes.io/projected/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-kube-api-access-p74q7\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.132353 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.132215    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.132353 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.132310    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.132483 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.132372    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.132483 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.132397    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.132483 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.132452    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.132872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.132784    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.133001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.132893    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.133077 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.133024    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.133196 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.133176    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b10d05c-ccd8-4125-819e-19e71a233aa9-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:29.144676 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.144002    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b10d05c-ccd8-4125-819e-19e71a233aa9-kube-api-access-p2rjm" pod "8b10d05c-ccd8-4125-819e-19e71a233aa9" (UID: "8b10d05c-ccd8-4125-819e-19e71a233aa9"). InnerVolumeSpecName "kube-api-access-p2rjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:46:29.234505 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.234427    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.234505 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.234465    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.234717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.234531    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.234717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.234564    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p74q7\" (UniqueName: \"kubernetes.io/projected/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-kube-api-access-p74q7\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.234717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.234591    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.234717 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.234670    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.234993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.234728    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.234993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.234753    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.234993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.234848    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.234993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.234965    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-credential-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.235203 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.235027    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-workload-certs\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.235203 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.235086    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-workload-socket\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.235203 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.235172    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-data\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.235357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.235218    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2rjm\" (UniqueName: \"kubernetes.io/projected/8b10d05c-ccd8-4125-819e-19e71a233aa9-kube-api-access-p2rjm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:29.235634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.235614    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istiod-ca-cert\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.237327 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.237294    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-envoy\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.237539 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.237521    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-podinfo\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.257777 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.257751    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-token\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.257910 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.257850    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p74q7\" (UniqueName: \"kubernetes.io/projected/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-kube-api-access-p74q7\") pod \"e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") " pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.333991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.333960    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:29.454171 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.454134    5428 generic.go:356] "Generic (PLEG): container finished" podID="8b10d05c-ccd8-4125-819e-19e71a233aa9" containerID="92d5122dcc6d92b3449f39354b078177d5e9d1ccf559028ef57df6b6a24844dc" exitCode=0
Oct 29 23:46:29.454373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.454180    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc"
Oct 29 23:46:29.454373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.454228    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc" event={"ID":"8b10d05c-ccd8-4125-819e-19e71a233aa9","Type":"ContainerDied","Data":"92d5122dcc6d92b3449f39354b078177d5e9d1ccf559028ef57df6b6a24844dc"}
Oct 29 23:46:29.454373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.454296    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc" event={"ID":"8b10d05c-ccd8-4125-819e-19e71a233aa9","Type":"ContainerDied","Data":"e6ac909b22e0b4b69282c59d38026c31e94f38065da3f0a53eae7d5934dc5358"}
Oct 29 23:46:29.454373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.454328    5428 scope.go:122] "RemoveContainer" containerID="92d5122dcc6d92b3449f39354b078177d5e9d1ccf559028ef57df6b6a24844dc"
Oct 29 23:46:29.469313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.469295    5428 scope.go:122] "RemoveContainer" containerID="92d5122dcc6d92b3449f39354b078177d5e9d1ccf559028ef57df6b6a24844dc"
Oct 29 23:46:29.469609 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:46:29.469587    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92d5122dcc6d92b3449f39354b078177d5e9d1ccf559028ef57df6b6a24844dc\": container with ID starting with 92d5122dcc6d92b3449f39354b078177d5e9d1ccf559028ef57df6b6a24844dc not found: ID does not exist" containerID="92d5122dcc6d92b3449f39354b078177d5e9d1ccf559028ef57df6b6a24844dc"
Oct 29 23:46:29.469689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.469616    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92d5122dcc6d92b3449f39354b078177d5e9d1ccf559028ef57df6b6a24844dc"} err="failed to get container status \"92d5122dcc6d92b3449f39354b078177d5e9d1ccf559028ef57df6b6a24844dc\": rpc error: code = NotFound desc = could not find container \"92d5122dcc6d92b3449f39354b078177d5e9d1ccf559028ef57df6b6a24844dc\": container with ID starting with 92d5122dcc6d92b3449f39354b078177d5e9d1ccf559028ef57df6b6a24844dc not found: ID does not exist"
Oct 29 23:46:29.510774 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.510693    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc"]
Oct 29 23:46:29.515465 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.515433    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7aeaad-79f94fd86c-rpthc"]
Oct 29 23:46:29.671418 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.671008    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"]
Oct 29 23:46:29.675972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.675933    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:46:29.676075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.676020    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:46:29.676075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:29.676060    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:46:30.032452 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:30.032370    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:30.467371 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:30.467335    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr" event={"ID":"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c","Type":"ContainerStarted","Data":"11f1d4be6b1f7bf59285dce0c10d3431ba5846455094582097409f5923cffbfd"}
Oct 29 23:46:30.467634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:30.467615    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr" event={"ID":"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c","Type":"ContainerStarted","Data":"4da447c210cbb214f8b7fdc257c303c832fa839fbc6e3d2364042722645794a4"}
Oct 29 23:46:30.552974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:30.552919    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr" podStartSLOduration=2.552904739 podStartE2EDuration="2.552904739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:46:30.540743362 +0000 UTC m=+3793.883692108" watchObservedRunningTime="2026-10-29 23:46:30.552904739 +0000 UTC m=+3793.895853484"
Oct 29 23:46:30.797311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:30.797205    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b10d05c-ccd8-4125-819e-19e71a233aa9" path="/var/lib/kubelet/pods/8b10d05c-ccd8-4125-819e-19e71a233aa9/volumes"
Oct 29 23:46:31.334673 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:31.334640    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:31.336250 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:31.336224    5428 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 29 23:46:31.336377 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:31.336271    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr" podUID="dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 29 23:46:31.788399 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:31.788355    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"]
Oct 29 23:46:31.788672 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:31.788641    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb" podUID="3555bd1b-d45c-4196-953d-af8ed8af9e63" containerName="maas-api" containerID="cri-o://7d0715accfeb3b27ef295776f7f45b5fb806db8e8b6fa01934b4a07b6be16efd" gracePeriod=30
Oct 29 23:46:31.885166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:31.885117    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:32.204188 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.204158    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"
Oct 29 23:46:32.309826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.307074    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3555bd1b-d45c-4196-953d-af8ed8af9e63-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3555bd1b-d45c-4196-953d-af8ed8af9e63-metrics-certs\") pod \"3555bd1b-d45c-4196-953d-af8ed8af9e63\" (UID: \"3555bd1b-d45c-4196-953d-af8ed8af9e63\") "
Oct 29 23:46:32.309826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.307183    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3555bd1b-d45c-4196-953d-af8ed8af9e63-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3555bd1b-d45c-4196-953d-af8ed8af9e63-maas-api-tls\") pod \"3555bd1b-d45c-4196-953d-af8ed8af9e63\" (UID: \"3555bd1b-d45c-4196-953d-af8ed8af9e63\") "
Oct 29 23:46:32.309826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.307263    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3555bd1b-d45c-4196-953d-af8ed8af9e63-kube-api-access-rbd5p\" (UniqueName: \"kubernetes.io/projected/3555bd1b-d45c-4196-953d-af8ed8af9e63-kube-api-access-rbd5p\") pod \"3555bd1b-d45c-4196-953d-af8ed8af9e63\" (UID: \"3555bd1b-d45c-4196-953d-af8ed8af9e63\") "
Oct 29 23:46:32.309826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.307343    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3555bd1b-d45c-4196-953d-af8ed8af9e63-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3555bd1b-d45c-4196-953d-af8ed8af9e63-openshift-service-ca\") pod \"3555bd1b-d45c-4196-953d-af8ed8af9e63\" (UID: \"3555bd1b-d45c-4196-953d-af8ed8af9e63\") "
Oct 29 23:46:32.320567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.312074    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3555bd1b-d45c-4196-953d-af8ed8af9e63-openshift-service-ca" pod "3555bd1b-d45c-4196-953d-af8ed8af9e63" (UID: "3555bd1b-d45c-4196-953d-af8ed8af9e63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:46:32.320567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.318991    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3555bd1b-d45c-4196-953d-af8ed8af9e63-kube-api-access-rbd5p" pod "3555bd1b-d45c-4196-953d-af8ed8af9e63" (UID: "3555bd1b-d45c-4196-953d-af8ed8af9e63"). InnerVolumeSpecName "kube-api-access-rbd5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:46:32.328820 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.320994    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3555bd1b-d45c-4196-953d-af8ed8af9e63-maas-api-tls" pod "3555bd1b-d45c-4196-953d-af8ed8af9e63" (UID: "3555bd1b-d45c-4196-953d-af8ed8af9e63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:46:32.328820 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.321342    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3555bd1b-d45c-4196-953d-af8ed8af9e63-metrics-certs" pod "3555bd1b-d45c-4196-953d-af8ed8af9e63" (UID: "3555bd1b-d45c-4196-953d-af8ed8af9e63"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:46:32.339266 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.336250    5428 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 29 23:46:32.339266 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.336303    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr" podUID="dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 29 23:46:32.412886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.412758    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3555bd1b-d45c-4196-953d-af8ed8af9e63-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:32.412886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.412845    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbd5p\" (UniqueName: \"kubernetes.io/projected/3555bd1b-d45c-4196-953d-af8ed8af9e63-kube-api-access-rbd5p\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:32.412886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.412864    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3555bd1b-d45c-4196-953d-af8ed8af9e63-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:32.412886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.412878    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3555bd1b-d45c-4196-953d-af8ed8af9e63-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:32.505524 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.505483    5428 generic.go:356] "Generic (PLEG): container finished" podID="3555bd1b-d45c-4196-953d-af8ed8af9e63" containerID="7d0715accfeb3b27ef295776f7f45b5fb806db8e8b6fa01934b4a07b6be16efd" exitCode=0
Oct 29 23:46:32.506419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.506079    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb" event={"ID":"3555bd1b-d45c-4196-953d-af8ed8af9e63","Type":"ContainerDied","Data":"7d0715accfeb3b27ef295776f7f45b5fb806db8e8b6fa01934b4a07b6be16efd"}
Oct 29 23:46:32.506419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.506097    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"
Oct 29 23:46:32.506419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.506144    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb" event={"ID":"3555bd1b-d45c-4196-953d-af8ed8af9e63","Type":"ContainerDied","Data":"d3ca839ed30616103ada2f11fd0b8de643a19302a38593f331ac3bfe189ddab8"}
Oct 29 23:46:32.506419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.506177    5428 scope.go:122] "RemoveContainer" containerID="7d0715accfeb3b27ef295776f7f45b5fb806db8e8b6fa01934b4a07b6be16efd"
Oct 29 23:46:32.519033 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.519015    5428 scope.go:122] "RemoveContainer" containerID="7d0715accfeb3b27ef295776f7f45b5fb806db8e8b6fa01934b4a07b6be16efd"
Oct 29 23:46:32.519361 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:46:32.519338    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d0715accfeb3b27ef295776f7f45b5fb806db8e8b6fa01934b4a07b6be16efd\": container with ID starting with 7d0715accfeb3b27ef295776f7f45b5fb806db8e8b6fa01934b4a07b6be16efd not found: ID does not exist" containerID="7d0715accfeb3b27ef295776f7f45b5fb806db8e8b6fa01934b4a07b6be16efd"
Oct 29 23:46:32.519408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.519372    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d0715accfeb3b27ef295776f7f45b5fb806db8e8b6fa01934b4a07b6be16efd"} err="failed to get container status \"7d0715accfeb3b27ef295776f7f45b5fb806db8e8b6fa01934b4a07b6be16efd\": rpc error: code = NotFound desc = could not find container \"7d0715accfeb3b27ef295776f7f45b5fb806db8e8b6fa01934b4a07b6be16efd\": container with ID starting with 7d0715accfeb3b27ef295776f7f45b5fb806db8e8b6fa01934b4a07b6be16efd not found: ID does not exist"
Oct 29 23:46:32.586828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.584995    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"]
Oct 29 23:46:32.590641 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.590618    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f162a-67d94f8544-vtmcb"]
Oct 29 23:46:32.609736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.609712    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:32.793143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.793065    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3555bd1b-d45c-4196-953d-af8ed8af9e63" path="/var/lib/kubelet/pods/3555bd1b-d45c-4196-953d-af8ed8af9e63/volumes"
Oct 29 23:46:32.906534 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:32.906499    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:33.239147 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:33.239103    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:33.298548 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:33.298517    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:33.311109 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:33.311082    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:33.334307 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:33.334273    5428 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 29 23:46:33.334616 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:33.334331    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr" podUID="dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 29 23:46:33.389658 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:33.387709    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:33.527094 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:33.527006    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:46:33.535435 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:33.535408    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:34.334293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:34.334259    5428 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 29 23:46:34.334520 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:34.334320    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr" podUID="dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 29 23:46:35.334641 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:35.334604    5428 patch_prober.go:29] interesting pod/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Oct 29 23:46:35.335129 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:35.334673    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr" podUID="dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Oct 29 23:46:35.997377 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:35.997342    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:36.183355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:36.183321    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:36.312392 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:36.312302    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:36.496087 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:36.496054    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:36.497235 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:36.497167    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:36.497941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:36.497915    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:46:36.656323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:36.656288    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:36.803336 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:36.803308    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:37.195644 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:37.195608    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:37.802216 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:37.802178    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:38.541226 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:38.541186    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:39.940592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:39.940557    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:40.061715 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:40.061680    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:40.293188 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:40.293099    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:40.507711 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:40.507680    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:42.573179 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:42.573139    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:43.875594 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:43.875559    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:44.010784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:44.010744    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:44.604853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:44.604818    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:46.395131 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:46.395095    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:47.376436 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:46:47.376393    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:46:48.152572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:48.152541    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq"]
Oct 29 23:46:48.188301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:48.188274    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:48.305764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:48.305734    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:48.431097 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:48.430999    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:50.195430 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:50.195396    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:51.302880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:51.302845    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd"]
Oct 29 23:46:51.335526 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:51.335489    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8"]
Oct 29 23:46:51.441602 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:51.441572    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:51.573933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:51.573842    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:52.005634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:52.005601    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:53.153606 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.153551    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq" podUID="3ab912fe-021d-406d-8120-c428d151f1d8" containerName="payload-pre-processing" containerID="cri-o://1f0ba8924c9d243ada5da71676ca42cbf0a76ad86bb106db7f70fa5ed00ad37a" gracePeriod=25
Oct 29 23:46:53.530647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.530537    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"]
Oct 29 23:46:53.530909 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.530861    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj" podUID="9e14ce23-ae46-4db5-9275-ceaf9987ba61" containerName="maas-api" containerID="cri-o://518ace8c14d11f4531e1abb2a19454bea5aa755fcc8e63376ee89c8381710104" gracePeriod=30
Oct 29 23:46:53.624726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.624693    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:53.692476 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.692442    5428 generic.go:356] "Generic (PLEG): container finished" podID="9e14ce23-ae46-4db5-9275-ceaf9987ba61" containerID="518ace8c14d11f4531e1abb2a19454bea5aa755fcc8e63376ee89c8381710104" exitCode=0
Oct 29 23:46:53.692651 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.692522    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj" event={"ID":"9e14ce23-ae46-4db5-9275-ceaf9987ba61","Type":"ContainerDied","Data":"518ace8c14d11f4531e1abb2a19454bea5aa755fcc8e63376ee89c8381710104"}
Oct 29 23:46:53.869533 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.869509    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:46:53.888669 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.888640    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e14ce23-ae46-4db5-9275-ceaf9987ba61-kube-api-access-bfths\" (UniqueName: \"kubernetes.io/projected/9e14ce23-ae46-4db5-9275-ceaf9987ba61-kube-api-access-bfths\") pod \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\" (UID: \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\") "
Oct 29 23:46:53.889479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.888675    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e14ce23-ae46-4db5-9275-ceaf9987ba61-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e14ce23-ae46-4db5-9275-ceaf9987ba61-maas-api-tls\") pod \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\" (UID: \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\") "
Oct 29 23:46:53.889479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.888727    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e14ce23-ae46-4db5-9275-ceaf9987ba61-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e14ce23-ae46-4db5-9275-ceaf9987ba61-openshift-service-ca\") pod \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\" (UID: \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\") "
Oct 29 23:46:53.889479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.888772    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e14ce23-ae46-4db5-9275-ceaf9987ba61-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e14ce23-ae46-4db5-9275-ceaf9987ba61-metrics-certs\") pod \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\" (UID: \"9e14ce23-ae46-4db5-9275-ceaf9987ba61\") "
Oct 29 23:46:53.889479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.889116    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e14ce23-ae46-4db5-9275-ceaf9987ba61-openshift-service-ca" pod "9e14ce23-ae46-4db5-9275-ceaf9987ba61" (UID: "9e14ce23-ae46-4db5-9275-ceaf9987ba61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:46:53.889753 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.889592    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e14ce23-ae46-4db5-9275-ceaf9987ba61-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:53.891004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.890977    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e14ce23-ae46-4db5-9275-ceaf9987ba61-kube-api-access-bfths" pod "9e14ce23-ae46-4db5-9275-ceaf9987ba61" (UID: "9e14ce23-ae46-4db5-9275-ceaf9987ba61"). InnerVolumeSpecName "kube-api-access-bfths". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:46:53.891118 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.891000    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e14ce23-ae46-4db5-9275-ceaf9987ba61-maas-api-tls" pod "9e14ce23-ae46-4db5-9275-ceaf9987ba61" (UID: "9e14ce23-ae46-4db5-9275-ceaf9987ba61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:46:53.891118 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.891022    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e14ce23-ae46-4db5-9275-ceaf9987ba61-metrics-certs" pod "9e14ce23-ae46-4db5-9275-ceaf9987ba61" (UID: "9e14ce23-ae46-4db5-9275-ceaf9987ba61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:46:53.991217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.991179    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfths\" (UniqueName: \"kubernetes.io/projected/9e14ce23-ae46-4db5-9275-ceaf9987ba61-kube-api-access-bfths\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:53.991217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.991213    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e14ce23-ae46-4db5-9275-ceaf9987ba61-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:53.991217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:53.991225    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e14ce23-ae46-4db5-9275-ceaf9987ba61-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:46:54.211708 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:54.211674    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:54.345548 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:54.345510    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq" podUID="3ab912fe-021d-406d-8120-c428d151f1d8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:46:54.448849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:54.448786    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:54.612985 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:54.612902    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:54.702050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:54.702017    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"
Oct 29 23:46:54.702050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:54.702035    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj" event={"ID":"9e14ce23-ae46-4db5-9275-ceaf9987ba61","Type":"ContainerDied","Data":"626e4450dc091c2896768a490aaf9c202820caadb12c78f1d026ef95d08edd4f"}
Oct 29 23:46:54.702338 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:54.702108    5428 scope.go:122] "RemoveContainer" containerID="518ace8c14d11f4531e1abb2a19454bea5aa755fcc8e63376ee89c8381710104"
Oct 29 23:46:54.766303 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:54.766276    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"]
Oct 29 23:46:54.779099 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:54.779066    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7d4110-7fbc7f5c68-gplpj"]
Oct 29 23:46:55.001380 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:55.001349    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:56.303682 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:56.303647    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd" podUID="9df13b19-5a4b-4eff-8733-66c06fdb38eb" containerName="payload-processing" containerID="cri-o://d35b7ebb92466f11b1884dc10ddacb4795175a42139fb8d917888052c8d5c46b" gracePeriod=25
Oct 29 23:46:56.336453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:56.336415    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8" podUID="4c1689ec-01d8-4ef6-885c-e15f332d06a0" containerName="payload-pre-processing" containerID="cri-o://1b6101dfb0177128d32818cfd80946618dfc19dc83bf7223ec7f89d3e317b70c" gracePeriod=25
Oct 29 23:46:56.389528 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:56.389498    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:46:56.780009 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:56.779970    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e14ce23-ae46-4db5-9275-ceaf9987ba61" path="/var/lib/kubelet/pods/9e14ce23-ae46-4db5-9275-ceaf9987ba61/volumes"
Oct 29 23:46:58.326029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:58.325993    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd" podUID="9df13b19-5a4b-4eff-8733-66c06fdb38eb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:46:58.326462 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:58.325989    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8" podUID="4c1689ec-01d8-4ef6-885c-e15f332d06a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:46:58.401163 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:46:58.401127    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:00.221420 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:00.221385    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:01.373691 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:01.373660    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:01.521595 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:01.521563    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:01.861578 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:01.861492    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:02.593358 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:02.593326    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:04.345378 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:04.345342    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq" podUID="3ab912fe-021d-406d-8120-c428d151f1d8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:47:04.610515 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:04.610435    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:06.554289 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:06.554226    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:08.325926 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:08.325879    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8" podUID="4c1689ec-01d8-4ef6-885c-e15f332d06a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:47:08.325926 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:08.325912    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd" podUID="9df13b19-5a4b-4eff-8733-66c06fdb38eb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:47:08.511481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:08.511445    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:10.479922 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:10.479891    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:12.291677 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:12.291643    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:12.824319 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:12.824284    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925"]
Oct 29 23:47:12.871544 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:12.871502    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58"]
Oct 29 23:47:12.962656 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:12.962604    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:13.076949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:13.076865    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:13.569867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:13.569842    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq"
Oct 29 23:47:13.667878 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:13.667843    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ab912fe-021d-406d-8120-c428d151f1d8-kube-api-access-bvqgc\" (UniqueName: \"kubernetes.io/projected/3ab912fe-021d-406d-8120-c428d151f1d8-kube-api-access-bvqgc\") pod \"3ab912fe-021d-406d-8120-c428d151f1d8\" (UID: \"3ab912fe-021d-406d-8120-c428d151f1d8\") "
Oct 29 23:47:13.668090 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:13.667889    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ab912fe-021d-406d-8120-c428d151f1d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ab912fe-021d-406d-8120-c428d151f1d8-plugins-config-volume\") pod \"3ab912fe-021d-406d-8120-c428d151f1d8\" (UID: \"3ab912fe-021d-406d-8120-c428d151f1d8\") "
Oct 29 23:47:13.668344 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:13.668318    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ab912fe-021d-406d-8120-c428d151f1d8-plugins-config-volume" pod "3ab912fe-021d-406d-8120-c428d151f1d8" (UID: "3ab912fe-021d-406d-8120-c428d151f1d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:47:13.668520 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:13.668503    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ab912fe-021d-406d-8120-c428d151f1d8-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:47:13.670007 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:13.669982    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab912fe-021d-406d-8120-c428d151f1d8-kube-api-access-bvqgc" pod "3ab912fe-021d-406d-8120-c428d151f1d8" (UID: "3ab912fe-021d-406d-8120-c428d151f1d8"). InnerVolumeSpecName "kube-api-access-bvqgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:47:13.769489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:13.769458    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvqgc\" (UniqueName: \"kubernetes.io/projected/3ab912fe-021d-406d-8120-c428d151f1d8-kube-api-access-bvqgc\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:47:13.870179 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:13.870145    5428 generic.go:356] "Generic (PLEG): container finished" podID="3ab912fe-021d-406d-8120-c428d151f1d8" containerID="1f0ba8924c9d243ada5da71676ca42cbf0a76ad86bb106db7f70fa5ed00ad37a" exitCode=0
Oct 29 23:47:13.870359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:13.870191    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq"
Oct 29 23:47:13.870359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:13.870228    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq" event={"ID":"3ab912fe-021d-406d-8120-c428d151f1d8","Type":"ContainerDied","Data":"1f0ba8924c9d243ada5da71676ca42cbf0a76ad86bb106db7f70fa5ed00ad37a"}
Oct 29 23:47:13.870359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:13.870293    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq" event={"ID":"3ab912fe-021d-406d-8120-c428d151f1d8","Type":"ContainerDied","Data":"70078a140edc491d80fc8e2d97f8cf1449462520ac2cadc99dab778edbc78207"}
Oct 29 23:47:13.870359 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:13.870317    5428 scope.go:122] "RemoveContainer" containerID="1f0ba8924c9d243ada5da71676ca42cbf0a76ad86bb106db7f70fa5ed00ad37a"
Oct 29 23:47:13.887675 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:13.887640    5428 scope.go:122] "RemoveContainer" containerID="1f0ba8924c9d243ada5da71676ca42cbf0a76ad86bb106db7f70fa5ed00ad37a"
Oct 29 23:47:13.888092 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:47:13.888056    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f0ba8924c9d243ada5da71676ca42cbf0a76ad86bb106db7f70fa5ed00ad37a\": container with ID starting with 1f0ba8924c9d243ada5da71676ca42cbf0a76ad86bb106db7f70fa5ed00ad37a not found: ID does not exist" containerID="1f0ba8924c9d243ada5da71676ca42cbf0a76ad86bb106db7f70fa5ed00ad37a"
Oct 29 23:47:13.888213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:13.888101    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f0ba8924c9d243ada5da71676ca42cbf0a76ad86bb106db7f70fa5ed00ad37a"} err="failed to get container status \"1f0ba8924c9d243ada5da71676ca42cbf0a76ad86bb106db7f70fa5ed00ad37a\": rpc error: code = NotFound desc = could not find container \"1f0ba8924c9d243ada5da71676ca42cbf0a76ad86bb106db7f70fa5ed00ad37a\": container with ID starting with 1f0ba8924c9d243ada5da71676ca42cbf0a76ad86bb106db7f70fa5ed00ad37a not found: ID does not exist"
Oct 29 23:47:13.923616 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:13.923547    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq"]
Oct 29 23:47:13.936145 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:13.936118    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b7a567-8649869f9f-tvgqq"]
Oct 29 23:47:14.284780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:14.284693    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:14.779375 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:14.779342    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ab912fe-021d-406d-8120-c428d151f1d8" path="/var/lib/kubelet/pods/3ab912fe-021d-406d-8120-c428d151f1d8/volumes"
Oct 29 23:47:16.124426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.124391    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:16.654316 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.654238    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd"
Oct 29 23:47:16.680053 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.680033    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8"
Oct 29 23:47:16.811108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.811079    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df13b19-5a4b-4eff-8733-66c06fdb38eb-kube-api-access-gt4ql\" (UniqueName: \"kubernetes.io/projected/9df13b19-5a4b-4eff-8733-66c06fdb38eb-kube-api-access-gt4ql\") pod \"9df13b19-5a4b-4eff-8733-66c06fdb38eb\" (UID: \"9df13b19-5a4b-4eff-8733-66c06fdb38eb\") "
Oct 29 23:47:16.811292 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.811114    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c1689ec-01d8-4ef6-885c-e15f332d06a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c1689ec-01d8-4ef6-885c-e15f332d06a0-plugins-config-volume\") pod \"4c1689ec-01d8-4ef6-885c-e15f332d06a0\" (UID: \"4c1689ec-01d8-4ef6-885c-e15f332d06a0\") "
Oct 29 23:47:16.811292 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.811242    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1689ec-01d8-4ef6-885c-e15f332d06a0-kube-api-access-49k7c\" (UniqueName: \"kubernetes.io/projected/4c1689ec-01d8-4ef6-885c-e15f332d06a0-kube-api-access-49k7c\") pod \"4c1689ec-01d8-4ef6-885c-e15f332d06a0\" (UID: \"4c1689ec-01d8-4ef6-885c-e15f332d06a0\") "
Oct 29 23:47:16.811436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.811386    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df13b19-5a4b-4eff-8733-66c06fdb38eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df13b19-5a4b-4eff-8733-66c06fdb38eb-plugins-config-volume\") pod \"9df13b19-5a4b-4eff-8733-66c06fdb38eb\" (UID: \"9df13b19-5a4b-4eff-8733-66c06fdb38eb\") "
Oct 29 23:47:16.811507 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.811457    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c1689ec-01d8-4ef6-885c-e15f332d06a0-plugins-config-volume" pod "4c1689ec-01d8-4ef6-885c-e15f332d06a0" (UID: "4c1689ec-01d8-4ef6-885c-e15f332d06a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:47:16.811842 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.811791    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df13b19-5a4b-4eff-8733-66c06fdb38eb-plugins-config-volume" pod "9df13b19-5a4b-4eff-8733-66c06fdb38eb" (UID: "9df13b19-5a4b-4eff-8733-66c06fdb38eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:47:16.812371 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.812346    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c1689ec-01d8-4ef6-885c-e15f332d06a0-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:47:16.812468 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.812376    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df13b19-5a4b-4eff-8733-66c06fdb38eb-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:47:16.813218 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.813197    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df13b19-5a4b-4eff-8733-66c06fdb38eb-kube-api-access-gt4ql" pod "9df13b19-5a4b-4eff-8733-66c06fdb38eb" (UID: "9df13b19-5a4b-4eff-8733-66c06fdb38eb"). InnerVolumeSpecName "kube-api-access-gt4ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:47:16.813470 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.813456    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1689ec-01d8-4ef6-885c-e15f332d06a0-kube-api-access-49k7c" pod "4c1689ec-01d8-4ef6-885c-e15f332d06a0" (UID: "4c1689ec-01d8-4ef6-885c-e15f332d06a0"). InnerVolumeSpecName "kube-api-access-49k7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:47:16.896360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.896330    5428 generic.go:356] "Generic (PLEG): container finished" podID="9df13b19-5a4b-4eff-8733-66c06fdb38eb" containerID="d35b7ebb92466f11b1884dc10ddacb4795175a42139fb8d917888052c8d5c46b" exitCode=0
Oct 29 23:47:16.896519 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.896378    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd"
Oct 29 23:47:16.896519 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.896415    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd" event={"ID":"9df13b19-5a4b-4eff-8733-66c06fdb38eb","Type":"ContainerDied","Data":"d35b7ebb92466f11b1884dc10ddacb4795175a42139fb8d917888052c8d5c46b"}
Oct 29 23:47:16.896519 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.896462    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd" event={"ID":"9df13b19-5a4b-4eff-8733-66c06fdb38eb","Type":"ContainerDied","Data":"6634a13d35e2be0efe8c02a49fab11936078bbd1d85d898978908407d2324d54"}
Oct 29 23:47:16.896519 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.896486    5428 scope.go:122] "RemoveContainer" containerID="d35b7ebb92466f11b1884dc10ddacb4795175a42139fb8d917888052c8d5c46b"
Oct 29 23:47:16.898059 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.898032    5428 generic.go:356] "Generic (PLEG): container finished" podID="4c1689ec-01d8-4ef6-885c-e15f332d06a0" containerID="1b6101dfb0177128d32818cfd80946618dfc19dc83bf7223ec7f89d3e317b70c" exitCode=0
Oct 29 23:47:16.898167 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.898072    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8"
Oct 29 23:47:16.898167 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.898110    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8" event={"ID":"4c1689ec-01d8-4ef6-885c-e15f332d06a0","Type":"ContainerDied","Data":"1b6101dfb0177128d32818cfd80946618dfc19dc83bf7223ec7f89d3e317b70c"}
Oct 29 23:47:16.898167 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.898161    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8" event={"ID":"4c1689ec-01d8-4ef6-885c-e15f332d06a0","Type":"ContainerDied","Data":"dd0a657adeed6201bb2dc9cb5e96024e6faad7fbd08fcac41de069cefa6addc8"}
Oct 29 23:47:16.909333 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.909315    5428 scope.go:122] "RemoveContainer" containerID="d35b7ebb92466f11b1884dc10ddacb4795175a42139fb8d917888052c8d5c46b"
Oct 29 23:47:16.909599 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:47:16.909582    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35b7ebb92466f11b1884dc10ddacb4795175a42139fb8d917888052c8d5c46b\": container with ID starting with d35b7ebb92466f11b1884dc10ddacb4795175a42139fb8d917888052c8d5c46b not found: ID does not exist" containerID="d35b7ebb92466f11b1884dc10ddacb4795175a42139fb8d917888052c8d5c46b"
Oct 29 23:47:16.909649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.909605    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35b7ebb92466f11b1884dc10ddacb4795175a42139fb8d917888052c8d5c46b"} err="failed to get container status \"d35b7ebb92466f11b1884dc10ddacb4795175a42139fb8d917888052c8d5c46b\": rpc error: code = NotFound desc = could not find container \"d35b7ebb92466f11b1884dc10ddacb4795175a42139fb8d917888052c8d5c46b\": container with ID starting with d35b7ebb92466f11b1884dc10ddacb4795175a42139fb8d917888052c8d5c46b not found: ID does not exist"
Oct 29 23:47:16.909649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.909621    5428 scope.go:122] "RemoveContainer" containerID="1b6101dfb0177128d32818cfd80946618dfc19dc83bf7223ec7f89d3e317b70c"
Oct 29 23:47:16.913868 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.913844    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt4ql\" (UniqueName: \"kubernetes.io/projected/9df13b19-5a4b-4eff-8733-66c06fdb38eb-kube-api-access-gt4ql\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:47:16.913975 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.913872    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49k7c\" (UniqueName: \"kubernetes.io/projected/4c1689ec-01d8-4ef6-885c-e15f332d06a0-kube-api-access-49k7c\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:47:16.921280 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.921255    5428 scope.go:122] "RemoveContainer" containerID="1b6101dfb0177128d32818cfd80946618dfc19dc83bf7223ec7f89d3e317b70c"
Oct 29 23:47:16.921564 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:47:16.921535    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b6101dfb0177128d32818cfd80946618dfc19dc83bf7223ec7f89d3e317b70c\": container with ID starting with 1b6101dfb0177128d32818cfd80946618dfc19dc83bf7223ec7f89d3e317b70c not found: ID does not exist" containerID="1b6101dfb0177128d32818cfd80946618dfc19dc83bf7223ec7f89d3e317b70c"
Oct 29 23:47:16.921624 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.921563    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6101dfb0177128d32818cfd80946618dfc19dc83bf7223ec7f89d3e317b70c"} err="failed to get container status \"1b6101dfb0177128d32818cfd80946618dfc19dc83bf7223ec7f89d3e317b70c\": rpc error: code = NotFound desc = could not find container \"1b6101dfb0177128d32818cfd80946618dfc19dc83bf7223ec7f89d3e317b70c\": container with ID starting with 1b6101dfb0177128d32818cfd80946618dfc19dc83bf7223ec7f89d3e317b70c not found: ID does not exist"
Oct 29 23:47:16.950333 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.950307    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd"]
Oct 29 23:47:16.954535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.954515    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fb367d-77c98894cd-l9zpd"]
Oct 29 23:47:16.989383 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.989354    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8"]
Oct 29 23:47:16.993398 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:16.993370    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fb367d-564db76f6c-pxnl8"]
Oct 29 23:47:17.370095 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:47:17.370051    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:47:17.820483 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:17.820445    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:17.825236 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:17.825202    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925" podUID="8fa0a50f-14ae-4ba8-a482-f7007672963c" containerName="payload-processing" containerID="cri-o://8d3514c2389c95ae6bd7d2c10383887bc7952d6b2e326e86f49e248a10ee38ab" gracePeriod=25
Oct 29 23:47:17.872193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:17.872153    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58" podUID="a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc" containerName="payload-pre-processing" containerID="cri-o://58ec4a565f3488512f709912ab88c9a8c13e36011b58c5ae4d16e6037c1ae057" gracePeriod=25
Oct 29 23:47:18.181276 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.181203    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925"
Oct 29 23:47:18.195318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.195296    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58"
Oct 29 23:47:18.335641 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.335611    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc-kube-api-access-jdzgr\" (UniqueName: \"kubernetes.io/projected/a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc-kube-api-access-jdzgr\") pod \"a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc\" (UID: \"a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc\") "
Oct 29 23:47:18.335827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.335659    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa0a50f-14ae-4ba8-a482-f7007672963c-kube-api-access-prbdn\" (UniqueName: \"kubernetes.io/projected/8fa0a50f-14ae-4ba8-a482-f7007672963c-kube-api-access-prbdn\") pod \"8fa0a50f-14ae-4ba8-a482-f7007672963c\" (UID: \"8fa0a50f-14ae-4ba8-a482-f7007672963c\") "
Oct 29 23:47:18.335827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.335733    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa0a50f-14ae-4ba8-a482-f7007672963c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fa0a50f-14ae-4ba8-a482-f7007672963c-plugins-config-volume\") pod \"8fa0a50f-14ae-4ba8-a482-f7007672963c\" (UID: \"8fa0a50f-14ae-4ba8-a482-f7007672963c\") "
Oct 29 23:47:18.336070 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.336045    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc-plugins-config-volume\") pod \"a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc\" (UID: \"a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc\") "
Oct 29 23:47:18.336214 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.336170    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa0a50f-14ae-4ba8-a482-f7007672963c-plugins-config-volume" pod "8fa0a50f-14ae-4ba8-a482-f7007672963c" (UID: "8fa0a50f-14ae-4ba8-a482-f7007672963c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:47:18.336698 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.336674    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc-plugins-config-volume" pod "a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc" (UID: "a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:47:18.336779 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.336760    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:47:18.336849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.336779    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fa0a50f-14ae-4ba8-a482-f7007672963c-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:47:18.338455 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.338423    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa0a50f-14ae-4ba8-a482-f7007672963c-kube-api-access-prbdn" pod "8fa0a50f-14ae-4ba8-a482-f7007672963c" (UID: "8fa0a50f-14ae-4ba8-a482-f7007672963c"). InnerVolumeSpecName "kube-api-access-prbdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:47:18.338579 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.338456    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc-kube-api-access-jdzgr" pod "a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc" (UID: "a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc"). InnerVolumeSpecName "kube-api-access-jdzgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:47:18.438379 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.438340    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdzgr\" (UniqueName: \"kubernetes.io/projected/a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc-kube-api-access-jdzgr\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:47:18.438379 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.438370    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prbdn\" (UniqueName: \"kubernetes.io/projected/8fa0a50f-14ae-4ba8-a482-f7007672963c-kube-api-access-prbdn\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:47:18.780173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.780075    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c1689ec-01d8-4ef6-885c-e15f332d06a0" path="/var/lib/kubelet/pods/4c1689ec-01d8-4ef6-885c-e15f332d06a0/volumes"
Oct 29 23:47:18.780726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.780705    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df13b19-5a4b-4eff-8733-66c06fdb38eb" path="/var/lib/kubelet/pods/9df13b19-5a4b-4eff-8733-66c06fdb38eb/volumes"
Oct 29 23:47:18.919463 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.919424    5428 generic.go:356] "Generic (PLEG): container finished" podID="a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc" containerID="58ec4a565f3488512f709912ab88c9a8c13e36011b58c5ae4d16e6037c1ae057" exitCode=0
Oct 29 23:47:18.919688 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.919474    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58"
Oct 29 23:47:18.919688 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.919511    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58" event={"ID":"a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc","Type":"ContainerDied","Data":"58ec4a565f3488512f709912ab88c9a8c13e36011b58c5ae4d16e6037c1ae057"}
Oct 29 23:47:18.919688 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.919580    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58" event={"ID":"a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc","Type":"ContainerDied","Data":"4d95471683cf5cb55bbbc14866a51983b3da4cfa60f4969fe08b55c10c70d73a"}
Oct 29 23:47:18.919688 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.919611    5428 scope.go:122] "RemoveContainer" containerID="58ec4a565f3488512f709912ab88c9a8c13e36011b58c5ae4d16e6037c1ae057"
Oct 29 23:47:18.921694 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.921668    5428 generic.go:356] "Generic (PLEG): container finished" podID="8fa0a50f-14ae-4ba8-a482-f7007672963c" containerID="8d3514c2389c95ae6bd7d2c10383887bc7952d6b2e326e86f49e248a10ee38ab" exitCode=0
Oct 29 23:47:18.921833 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.921706    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925" event={"ID":"8fa0a50f-14ae-4ba8-a482-f7007672963c","Type":"ContainerDied","Data":"8d3514c2389c95ae6bd7d2c10383887bc7952d6b2e326e86f49e248a10ee38ab"}
Oct 29 23:47:18.921833 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.921739    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925"
Oct 29 23:47:18.922047 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.921740    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925" event={"ID":"8fa0a50f-14ae-4ba8-a482-f7007672963c","Type":"ContainerDied","Data":"1816aebff5fbb4b194ca63eb75f6208d89c7cf2ee6b687da7c0c5c73f1f97fde"}
Oct 29 23:47:18.934715 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.934694    5428 scope.go:122] "RemoveContainer" containerID="58ec4a565f3488512f709912ab88c9a8c13e36011b58c5ae4d16e6037c1ae057"
Oct 29 23:47:18.935037 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:47:18.935017    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ec4a565f3488512f709912ab88c9a8c13e36011b58c5ae4d16e6037c1ae057\": container with ID starting with 58ec4a565f3488512f709912ab88c9a8c13e36011b58c5ae4d16e6037c1ae057 not found: ID does not exist" containerID="58ec4a565f3488512f709912ab88c9a8c13e36011b58c5ae4d16e6037c1ae057"
Oct 29 23:47:18.935139 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.935048    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ec4a565f3488512f709912ab88c9a8c13e36011b58c5ae4d16e6037c1ae057"} err="failed to get container status \"58ec4a565f3488512f709912ab88c9a8c13e36011b58c5ae4d16e6037c1ae057\": rpc error: code = NotFound desc = could not find container \"58ec4a565f3488512f709912ab88c9a8c13e36011b58c5ae4d16e6037c1ae057\": container with ID starting with 58ec4a565f3488512f709912ab88c9a8c13e36011b58c5ae4d16e6037c1ae057 not found: ID does not exist"
Oct 29 23:47:18.935139 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.935072    5428 scope.go:122] "RemoveContainer" containerID="8d3514c2389c95ae6bd7d2c10383887bc7952d6b2e326e86f49e248a10ee38ab"
Oct 29 23:47:18.946698 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.946678    5428 scope.go:122] "RemoveContainer" containerID="8d3514c2389c95ae6bd7d2c10383887bc7952d6b2e326e86f49e248a10ee38ab"
Oct 29 23:47:18.947040 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:47:18.947013    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d3514c2389c95ae6bd7d2c10383887bc7952d6b2e326e86f49e248a10ee38ab\": container with ID starting with 8d3514c2389c95ae6bd7d2c10383887bc7952d6b2e326e86f49e248a10ee38ab not found: ID does not exist" containerID="8d3514c2389c95ae6bd7d2c10383887bc7952d6b2e326e86f49e248a10ee38ab"
Oct 29 23:47:18.947278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.947046    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d3514c2389c95ae6bd7d2c10383887bc7952d6b2e326e86f49e248a10ee38ab"} err="failed to get container status \"8d3514c2389c95ae6bd7d2c10383887bc7952d6b2e326e86f49e248a10ee38ab\": rpc error: code = NotFound desc = could not find container \"8d3514c2389c95ae6bd7d2c10383887bc7952d6b2e326e86f49e248a10ee38ab\": container with ID starting with 8d3514c2389c95ae6bd7d2c10383887bc7952d6b2e326e86f49e248a10ee38ab not found: ID does not exist"
Oct 29 23:47:18.973236 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.973210    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58"]
Oct 29 23:47:18.986471 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:18.986440    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b4b5e3-76dc6cbc99-q4w58"]
Oct 29 23:47:19.011957 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:19.011931    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925"]
Oct 29 23:47:19.018381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:19.018346    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b4b5e3-7b859ff4c8-8x925"]
Oct 29 23:47:19.593216 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:19.593174    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:20.778457 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:20.778419    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa0a50f-14ae-4ba8-a482-f7007672963c" path="/var/lib/kubelet/pods/8fa0a50f-14ae-4ba8-a482-f7007672963c/volumes"
Oct 29 23:47:20.778982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:20.778870    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc" path="/var/lib/kubelet/pods/a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc/volumes"
Oct 29 23:47:21.404925 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:21.404881    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:23.161276 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:23.161240    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:24.893127 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:24.893101    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:26.789629 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:26.789586    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:28.576486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:28.576451    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:30.900008 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:30.899975    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:32.706287 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:32.706251    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:33.342543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:33.342511    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:34.149087 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:34.149037    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:34.485932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:34.485844    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:36.277724 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:36.277692    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:38.210646 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:38.210616    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:40.102959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:40.102924    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:41.898740 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:41.898708    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:43.408225 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:43.408184    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"]
Oct 29 23:47:43.408729 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:43.408439    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72" podUID="b546f7f7-0c2c-4da9-a00d-41b52bb85815" containerName="maas-api" containerID="cri-o://20b4af7b06c5d36f49638f68e7e66f7c47531c18b5baec3a56ef194c8d6090cb" gracePeriod=30
Oct 29 23:47:43.500494 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:43.500454    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:43.733179 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:43.733149    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"
Oct 29 23:47:43.823490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:43.823455    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b546f7f7-0c2c-4da9-a00d-41b52bb85815-kube-api-access-ghbpr\" (UniqueName: \"kubernetes.io/projected/b546f7f7-0c2c-4da9-a00d-41b52bb85815-kube-api-access-ghbpr\") pod \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\" (UID: \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\") "
Oct 29 23:47:43.823696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:43.823533    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b546f7f7-0c2c-4da9-a00d-41b52bb85815-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b546f7f7-0c2c-4da9-a00d-41b52bb85815-metrics-certs\") pod \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\" (UID: \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\") "
Oct 29 23:47:43.823696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:43.823671    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b546f7f7-0c2c-4da9-a00d-41b52bb85815-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b546f7f7-0c2c-4da9-a00d-41b52bb85815-openshift-service-ca\") pod \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\" (UID: \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\") "
Oct 29 23:47:43.823863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:43.823776    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b546f7f7-0c2c-4da9-a00d-41b52bb85815-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b546f7f7-0c2c-4da9-a00d-41b52bb85815-maas-api-tls\") pod \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\" (UID: \"b546f7f7-0c2c-4da9-a00d-41b52bb85815\") "
Oct 29 23:47:43.824108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:43.824083    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b546f7f7-0c2c-4da9-a00d-41b52bb85815-openshift-service-ca" pod "b546f7f7-0c2c-4da9-a00d-41b52bb85815" (UID: "b546f7f7-0c2c-4da9-a00d-41b52bb85815"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:47:43.824727 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:43.824638    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b546f7f7-0c2c-4da9-a00d-41b52bb85815-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:47:43.825891 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:43.825860    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b546f7f7-0c2c-4da9-a00d-41b52bb85815-maas-api-tls" pod "b546f7f7-0c2c-4da9-a00d-41b52bb85815" (UID: "b546f7f7-0c2c-4da9-a00d-41b52bb85815"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:47:43.826183 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:43.826161    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b546f7f7-0c2c-4da9-a00d-41b52bb85815-metrics-certs" pod "b546f7f7-0c2c-4da9-a00d-41b52bb85815" (UID: "b546f7f7-0c2c-4da9-a00d-41b52bb85815"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:47:43.828534 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:43.828511    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b546f7f7-0c2c-4da9-a00d-41b52bb85815-kube-api-access-ghbpr" pod "b546f7f7-0c2c-4da9-a00d-41b52bb85815" (UID: "b546f7f7-0c2c-4da9-a00d-41b52bb85815"). InnerVolumeSpecName "kube-api-access-ghbpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:47:43.926988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:43.926898    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b546f7f7-0c2c-4da9-a00d-41b52bb85815-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:47:43.926988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:43.926937    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b546f7f7-0c2c-4da9-a00d-41b52bb85815-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:47:43.926988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:43.926952    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghbpr\" (UniqueName: \"kubernetes.io/projected/b546f7f7-0c2c-4da9-a00d-41b52bb85815-kube-api-access-ghbpr\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:47:44.178935 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:44.178834    5428 generic.go:356] "Generic (PLEG): container finished" podID="b546f7f7-0c2c-4da9-a00d-41b52bb85815" containerID="20b4af7b06c5d36f49638f68e7e66f7c47531c18b5baec3a56ef194c8d6090cb" exitCode=0
Oct 29 23:47:44.178935 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:44.178895    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"
Oct 29 23:47:44.179164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:44.178944    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72" event={"ID":"b546f7f7-0c2c-4da9-a00d-41b52bb85815","Type":"ContainerDied","Data":"20b4af7b06c5d36f49638f68e7e66f7c47531c18b5baec3a56ef194c8d6090cb"}
Oct 29 23:47:44.179164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:44.179012    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72" event={"ID":"b546f7f7-0c2c-4da9-a00d-41b52bb85815","Type":"ContainerDied","Data":"56e272e861491919b4370fc94e9e5e2b3970ca7f91643a68d37b8dedc4f2ef3e"}
Oct 29 23:47:44.179164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:44.179042    5428 scope.go:122] "RemoveContainer" containerID="20b4af7b06c5d36f49638f68e7e66f7c47531c18b5baec3a56ef194c8d6090cb"
Oct 29 23:47:44.194965 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:44.194942    5428 scope.go:122] "RemoveContainer" containerID="20b4af7b06c5d36f49638f68e7e66f7c47531c18b5baec3a56ef194c8d6090cb"
Oct 29 23:47:44.195409 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:47:44.195382    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b4af7b06c5d36f49638f68e7e66f7c47531c18b5baec3a56ef194c8d6090cb\": container with ID starting with 20b4af7b06c5d36f49638f68e7e66f7c47531c18b5baec3a56ef194c8d6090cb not found: ID does not exist" containerID="20b4af7b06c5d36f49638f68e7e66f7c47531c18b5baec3a56ef194c8d6090cb"
Oct 29 23:47:44.195529 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:44.195421    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b4af7b06c5d36f49638f68e7e66f7c47531c18b5baec3a56ef194c8d6090cb"} err="failed to get container status \"20b4af7b06c5d36f49638f68e7e66f7c47531c18b5baec3a56ef194c8d6090cb\": rpc error: code = NotFound desc = could not find container \"20b4af7b06c5d36f49638f68e7e66f7c47531c18b5baec3a56ef194c8d6090cb\": container with ID starting with 20b4af7b06c5d36f49638f68e7e66f7c47531c18b5baec3a56ef194c8d6090cb not found: ID does not exist"
Oct 29 23:47:44.235152 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:44.235123    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"]
Oct 29 23:47:44.239919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:44.239893    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb367d-644876f66c-dbl72"]
Oct 29 23:47:44.263458 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:44.263430    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:44.433206 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:44.433101    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:44.565926 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:44.565893    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:44.778666 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:44.778589    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b546f7f7-0c2c-4da9-a00d-41b52bb85815" path="/var/lib/kubelet/pods/b546f7f7-0c2c-4da9-a00d-41b52bb85815/volumes"
Oct 29 23:47:44.963671 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:44.963634    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:46.073433 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:46.073384    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:46.830029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:46.829985    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:47.057791 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:47.057762    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:47.277534 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:47.277498    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:47.369070 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:47:47.369023    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:47:47.628371 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:47.628337    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:49.256108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:49.256071    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:51.097824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:51.090068    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:51.306785 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:51.306752    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:51.441122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:51.441091    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:51.639255 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:51.639228    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:53.309262 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:53.309220    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:54.149729 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:54.149700    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:54.273951 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:54.273864    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:54.525939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:54.525794    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:55.698255 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:55.698216    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:47:57.851724 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:47:57.851649    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:00.053249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:00.053215    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:01.945789 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:01.945757    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:02.730135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:02.730099    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29"]
Oct 29 23:48:02.760862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:02.760831    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58"]
Oct 29 23:48:02.904855 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:02.904770    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:03.028030 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:03.027944    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:04.019034 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:04.018996    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:05.517188 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:05.517149    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"]
Oct 29 23:48:05.517791 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:05.517375    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq" podUID="92936ba4-7ef1-451e-9322-7cf3583600f5" containerName="maas-api" containerID="cri-o://97fe32b57861dad5bb1999c67c5d0d347fd77896bf746c321f4998c3b2ae8f59" gracePeriod=30
Oct 29 23:48:05.579052 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:05.579016    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:05.690520 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:05.690492    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:05.806572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:05.806544    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"
Oct 29 23:48:05.895510 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:05.895478    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92936ba4-7ef1-451e-9322-7cf3583600f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92936ba4-7ef1-451e-9322-7cf3583600f5-metrics-certs\") pod \"92936ba4-7ef1-451e-9322-7cf3583600f5\" (UID: \"92936ba4-7ef1-451e-9322-7cf3583600f5\") "
Oct 29 23:48:05.895698 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:05.895541    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92936ba4-7ef1-451e-9322-7cf3583600f5-kube-api-access-rf9kv\" (UniqueName: \"kubernetes.io/projected/92936ba4-7ef1-451e-9322-7cf3583600f5-kube-api-access-rf9kv\") pod \"92936ba4-7ef1-451e-9322-7cf3583600f5\" (UID: \"92936ba4-7ef1-451e-9322-7cf3583600f5\") "
Oct 29 23:48:05.895698 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:05.895663    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92936ba4-7ef1-451e-9322-7cf3583600f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92936ba4-7ef1-451e-9322-7cf3583600f5-maas-api-tls\") pod \"92936ba4-7ef1-451e-9322-7cf3583600f5\" (UID: \"92936ba4-7ef1-451e-9322-7cf3583600f5\") "
Oct 29 23:48:05.895865 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:05.895779    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92936ba4-7ef1-451e-9322-7cf3583600f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92936ba4-7ef1-451e-9322-7cf3583600f5-openshift-service-ca\") pod \"92936ba4-7ef1-451e-9322-7cf3583600f5\" (UID: \"92936ba4-7ef1-451e-9322-7cf3583600f5\") "
Oct 29 23:48:05.896792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:05.896288    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92936ba4-7ef1-451e-9322-7cf3583600f5-openshift-service-ca" pod "92936ba4-7ef1-451e-9322-7cf3583600f5" (UID: "92936ba4-7ef1-451e-9322-7cf3583600f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:48:05.897102 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:05.897072    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92936ba4-7ef1-451e-9322-7cf3583600f5-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:05.898316 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:05.898289    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92936ba4-7ef1-451e-9322-7cf3583600f5-metrics-certs" pod "92936ba4-7ef1-451e-9322-7cf3583600f5" (UID: "92936ba4-7ef1-451e-9322-7cf3583600f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:48:05.898423 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:05.898384    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92936ba4-7ef1-451e-9322-7cf3583600f5-kube-api-access-rf9kv" pod "92936ba4-7ef1-451e-9322-7cf3583600f5" (UID: "92936ba4-7ef1-451e-9322-7cf3583600f5"). InnerVolumeSpecName "kube-api-access-rf9kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:48:05.899114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:05.899094    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92936ba4-7ef1-451e-9322-7cf3583600f5-maas-api-tls" pod "92936ba4-7ef1-451e-9322-7cf3583600f5" (UID: "92936ba4-7ef1-451e-9322-7cf3583600f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:48:05.998568 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:05.998539    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92936ba4-7ef1-451e-9322-7cf3583600f5-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:05.998568 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:05.998566    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf9kv\" (UniqueName: \"kubernetes.io/projected/92936ba4-7ef1-451e-9322-7cf3583600f5-kube-api-access-rf9kv\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:05.998734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:05.998577    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92936ba4-7ef1-451e-9322-7cf3583600f5-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:06.190823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:06.190206    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:06.356687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:06.356653    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:06.452695 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:06.452617    5428 generic.go:356] "Generic (PLEG): container finished" podID="92936ba4-7ef1-451e-9322-7cf3583600f5" containerID="97fe32b57861dad5bb1999c67c5d0d347fd77896bf746c321f4998c3b2ae8f59" exitCode=0
Oct 29 23:48:06.452695 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:06.452660    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"
Oct 29 23:48:06.452919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:06.452693    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq" event={"ID":"92936ba4-7ef1-451e-9322-7cf3583600f5","Type":"ContainerDied","Data":"97fe32b57861dad5bb1999c67c5d0d347fd77896bf746c321f4998c3b2ae8f59"}
Oct 29 23:48:06.452919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:06.452745    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq" event={"ID":"92936ba4-7ef1-451e-9322-7cf3583600f5","Type":"ContainerDied","Data":"1ddec55a416fddf86eb07027a0d100236de637517b161364b51a6a277b5e8d66"}
Oct 29 23:48:06.452919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:06.452768    5428 scope.go:122] "RemoveContainer" containerID="97fe32b57861dad5bb1999c67c5d0d347fd77896bf746c321f4998c3b2ae8f59"
Oct 29 23:48:06.465853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:06.465836    5428 scope.go:122] "RemoveContainer" containerID="97fe32b57861dad5bb1999c67c5d0d347fd77896bf746c321f4998c3b2ae8f59"
Oct 29 23:48:06.466108 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:48:06.466089    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97fe32b57861dad5bb1999c67c5d0d347fd77896bf746c321f4998c3b2ae8f59\": container with ID starting with 97fe32b57861dad5bb1999c67c5d0d347fd77896bf746c321f4998c3b2ae8f59 not found: ID does not exist" containerID="97fe32b57861dad5bb1999c67c5d0d347fd77896bf746c321f4998c3b2ae8f59"
Oct 29 23:48:06.466172 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:06.466117    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97fe32b57861dad5bb1999c67c5d0d347fd77896bf746c321f4998c3b2ae8f59"} err="failed to get container status \"97fe32b57861dad5bb1999c67c5d0d347fd77896bf746c321f4998c3b2ae8f59\": rpc error: code = NotFound desc = could not find container \"97fe32b57861dad5bb1999c67c5d0d347fd77896bf746c321f4998c3b2ae8f59\": container with ID starting with 97fe32b57861dad5bb1999c67c5d0d347fd77896bf746c321f4998c3b2ae8f59 not found: ID does not exist"
Oct 29 23:48:06.512545 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:06.512496    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"]
Oct 29 23:48:06.512864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:06.512670    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:06.527937 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:06.527904    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b4b5e3-7b98b5cc49-xv7nq"]
Oct 29 23:48:06.778959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:06.778879    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92936ba4-7ef1-451e-9322-7cf3583600f5" path="/var/lib/kubelet/pods/92936ba4-7ef1-451e-9322-7cf3583600f5/volumes"
Oct 29 23:48:07.005022 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:07.004992    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:07.731338 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:07.731300    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29" podUID="103e3adb-1b76-44e2-ba11-03d11fea6b70" containerName="payload-processing" containerID="cri-o://ffe14db32e335551daa2be51aec2bcd36e4f081028c37b0e9b0db0a7aff3eb00" gracePeriod=25
Oct 29 23:48:07.761941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:07.761908    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58" podUID="e990be60-57d3-49f5-961a-d40381905773" containerName="payload-pre-processing" containerID="cri-o://33238e783b51fee2b7ffb3496f342d8142d6b458fd743b2fc802e337710e0003" gracePeriod=25
Oct 29 23:48:08.497023 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:08.496990    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:10.476571 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:10.476538    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:12.357020 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:12.356987    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:13.518039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:13.518004    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:13.707762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:13.707714    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:13.847810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:13.847718    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"]
Oct 29 23:48:13.848018 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:13.847992    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf" podUID="5603bb2b-7fee-4663-9f2d-9a550a7cca59" containerName="maas-api" containerID="cri-o://fa13b71238612875c10616b8c30b2f67798dc1200bf78d2b888caba44bf64a6e" gracePeriod=30
Oct 29 23:48:13.877403 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:13.877357    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:14.005035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.004998    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:14.136415 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.135012    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"
Oct 29 23:48:14.266448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.235094    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5603bb2b-7fee-4663-9f2d-9a550a7cca59-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5603bb2b-7fee-4663-9f2d-9a550a7cca59-openshift-service-ca\") pod \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\" (UID: \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\") "
Oct 29 23:48:14.266448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.235240    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5603bb2b-7fee-4663-9f2d-9a550a7cca59-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5603bb2b-7fee-4663-9f2d-9a550a7cca59-metrics-certs\") pod \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\" (UID: \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\") "
Oct 29 23:48:14.266448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.235441    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5603bb2b-7fee-4663-9f2d-9a550a7cca59-kube-api-access-bj6l6\" (UniqueName: \"kubernetes.io/projected/5603bb2b-7fee-4663-9f2d-9a550a7cca59-kube-api-access-bj6l6\") pod \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\" (UID: \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\") "
Oct 29 23:48:14.266448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.235592    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5603bb2b-7fee-4663-9f2d-9a550a7cca59-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5603bb2b-7fee-4663-9f2d-9a550a7cca59-maas-api-tls\") pod \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\" (UID: \"5603bb2b-7fee-4663-9f2d-9a550a7cca59\") "
Oct 29 23:48:14.266448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.238295    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5603bb2b-7fee-4663-9f2d-9a550a7cca59-openshift-service-ca" pod "5603bb2b-7fee-4663-9f2d-9a550a7cca59" (UID: "5603bb2b-7fee-4663-9f2d-9a550a7cca59"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:48:14.266448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.242007    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5603bb2b-7fee-4663-9f2d-9a550a7cca59-kube-api-access-bj6l6" pod "5603bb2b-7fee-4663-9f2d-9a550a7cca59" (UID: "5603bb2b-7fee-4663-9f2d-9a550a7cca59"). InnerVolumeSpecName "kube-api-access-bj6l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:48:14.266448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.242224    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5603bb2b-7fee-4663-9f2d-9a550a7cca59-metrics-certs" pod "5603bb2b-7fee-4663-9f2d-9a550a7cca59" (UID: "5603bb2b-7fee-4663-9f2d-9a550a7cca59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:48:14.266448 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.249202    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5603bb2b-7fee-4663-9f2d-9a550a7cca59-maas-api-tls" pod "5603bb2b-7fee-4663-9f2d-9a550a7cca59" (UID: "5603bb2b-7fee-4663-9f2d-9a550a7cca59"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:48:14.337586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.337552    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5603bb2b-7fee-4663-9f2d-9a550a7cca59-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:14.337586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.337584    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5603bb2b-7fee-4663-9f2d-9a550a7cca59-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:14.337586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.337594    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj6l6\" (UniqueName: \"kubernetes.io/projected/5603bb2b-7fee-4663-9f2d-9a550a7cca59-kube-api-access-bj6l6\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:14.337849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.337603    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5603bb2b-7fee-4663-9f2d-9a550a7cca59-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:14.364027 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.363988    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"]
Oct 29 23:48:14.364262 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.364238    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f" podUID="cf20849b-2d40-42ca-82fa-a91198bb012b" containerName="maas-api" containerID="cri-o://fb5c1d8b8db505ed68fcd767ed628c29626a31b55d520fcadf9efaddeb3b1ee4" gracePeriod=30
Oct 29 23:48:14.459823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.456285    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:14.586404 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.586351    5428 generic.go:356] "Generic (PLEG): container finished" podID="cf20849b-2d40-42ca-82fa-a91198bb012b" containerID="fb5c1d8b8db505ed68fcd767ed628c29626a31b55d520fcadf9efaddeb3b1ee4" exitCode=0
Oct 29 23:48:14.587089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.586736    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f" event={"ID":"cf20849b-2d40-42ca-82fa-a91198bb012b","Type":"ContainerDied","Data":"fb5c1d8b8db505ed68fcd767ed628c29626a31b55d520fcadf9efaddeb3b1ee4"}
Oct 29 23:48:14.589097 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.589062    5428 generic.go:356] "Generic (PLEG): container finished" podID="5603bb2b-7fee-4663-9f2d-9a550a7cca59" containerID="fa13b71238612875c10616b8c30b2f67798dc1200bf78d2b888caba44bf64a6e" exitCode=0
Oct 29 23:48:14.589265 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.589129    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf" event={"ID":"5603bb2b-7fee-4663-9f2d-9a550a7cca59","Type":"ContainerDied","Data":"fa13b71238612875c10616b8c30b2f67798dc1200bf78d2b888caba44bf64a6e"}
Oct 29 23:48:14.589265 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.589162    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf" event={"ID":"5603bb2b-7fee-4663-9f2d-9a550a7cca59","Type":"ContainerDied","Data":"ff8cbd091e2e89c8e69e420fdf6c34fac856632b75221ad40d2b9c47874291f7"}
Oct 29 23:48:14.589265 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.589183    5428 scope.go:122] "RemoveContainer" containerID="fa13b71238612875c10616b8c30b2f67798dc1200bf78d2b888caba44bf64a6e"
Oct 29 23:48:14.589443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.589299    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"
Oct 29 23:48:14.656921 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.656891    5428 scope.go:122] "RemoveContainer" containerID="fa13b71238612875c10616b8c30b2f67798dc1200bf78d2b888caba44bf64a6e"
Oct 29 23:48:14.658428 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:48:14.658357    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa13b71238612875c10616b8c30b2f67798dc1200bf78d2b888caba44bf64a6e\": container with ID starting with fa13b71238612875c10616b8c30b2f67798dc1200bf78d2b888caba44bf64a6e not found: ID does not exist" containerID="fa13b71238612875c10616b8c30b2f67798dc1200bf78d2b888caba44bf64a6e"
Oct 29 23:48:14.658428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.658400    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa13b71238612875c10616b8c30b2f67798dc1200bf78d2b888caba44bf64a6e"} err="failed to get container status \"fa13b71238612875c10616b8c30b2f67798dc1200bf78d2b888caba44bf64a6e\": rpc error: code = NotFound desc = could not find container \"fa13b71238612875c10616b8c30b2f67798dc1200bf78d2b888caba44bf64a6e\": container with ID starting with fa13b71238612875c10616b8c30b2f67798dc1200bf78d2b888caba44bf64a6e not found: ID does not exist"
Oct 29 23:48:14.659441 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.659416    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"]
Oct 29 23:48:14.668233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.668056    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-dea9cb-6d9d4cf5fd-sgwjf"]
Oct 29 23:48:14.681133 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.681113    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 23:48:14.745284 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.745250    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf20849b-2d40-42ca-82fa-a91198bb012b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf20849b-2d40-42ca-82fa-a91198bb012b-metrics-certs\") pod \"cf20849b-2d40-42ca-82fa-a91198bb012b\" (UID: \"cf20849b-2d40-42ca-82fa-a91198bb012b\") "
Oct 29 23:48:14.745489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.745314    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf20849b-2d40-42ca-82fa-a91198bb012b-kube-api-access-8dsnx\" (UniqueName: \"kubernetes.io/projected/cf20849b-2d40-42ca-82fa-a91198bb012b-kube-api-access-8dsnx\") pod \"cf20849b-2d40-42ca-82fa-a91198bb012b\" (UID: \"cf20849b-2d40-42ca-82fa-a91198bb012b\") "
Oct 29 23:48:14.745489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.745389    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf20849b-2d40-42ca-82fa-a91198bb012b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf20849b-2d40-42ca-82fa-a91198bb012b-openshift-service-ca\") pod \"cf20849b-2d40-42ca-82fa-a91198bb012b\" (UID: \"cf20849b-2d40-42ca-82fa-a91198bb012b\") "
Oct 29 23:48:14.745489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.745468    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf20849b-2d40-42ca-82fa-a91198bb012b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf20849b-2d40-42ca-82fa-a91198bb012b-maas-api-tls\") pod \"cf20849b-2d40-42ca-82fa-a91198bb012b\" (UID: \"cf20849b-2d40-42ca-82fa-a91198bb012b\") "
Oct 29 23:48:14.746122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.746085    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf20849b-2d40-42ca-82fa-a91198bb012b-openshift-service-ca" pod "cf20849b-2d40-42ca-82fa-a91198bb012b" (UID: "cf20849b-2d40-42ca-82fa-a91198bb012b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:48:14.746446 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.746421    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf20849b-2d40-42ca-82fa-a91198bb012b-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:14.748078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.748048    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf20849b-2d40-42ca-82fa-a91198bb012b-kube-api-access-8dsnx" pod "cf20849b-2d40-42ca-82fa-a91198bb012b" (UID: "cf20849b-2d40-42ca-82fa-a91198bb012b"). InnerVolumeSpecName "kube-api-access-8dsnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:48:14.748344 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.748322    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf20849b-2d40-42ca-82fa-a91198bb012b-maas-api-tls" pod "cf20849b-2d40-42ca-82fa-a91198bb012b" (UID: "cf20849b-2d40-42ca-82fa-a91198bb012b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:48:14.748460 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.748436    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf20849b-2d40-42ca-82fa-a91198bb012b-metrics-certs" pod "cf20849b-2d40-42ca-82fa-a91198bb012b" (UID: "cf20849b-2d40-42ca-82fa-a91198bb012b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:48:14.779589 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.779555    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5603bb2b-7fee-4663-9f2d-9a550a7cca59" path="/var/lib/kubelet/pods/5603bb2b-7fee-4663-9f2d-9a550a7cca59/volumes"
Oct 29 23:48:14.848454 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.848414    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dsnx\" (UniqueName: \"kubernetes.io/projected/cf20849b-2d40-42ca-82fa-a91198bb012b-kube-api-access-8dsnx\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:14.848454 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.848445    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf20849b-2d40-42ca-82fa-a91198bb012b-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:14.848454 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:14.848455    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf20849b-2d40-42ca-82fa-a91198bb012b-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:15.149067 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:15.149023    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:15.355090 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:15.355055    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:15.547889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:15.547780    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:15.598454 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:15.598423    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"
Oct 29 23:48:15.598953 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:15.598444    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f" event={"ID":"cf20849b-2d40-42ca-82fa-a91198bb012b","Type":"ContainerDied","Data":"358f98366ccac8d23fa58fc88145b171448320315069b01c81e7c5b3e3e8ce3d"}
Oct 29 23:48:15.598953 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:15.598612    5428 scope.go:122] "RemoveContainer" containerID="fb5c1d8b8db505ed68fcd767ed628c29626a31b55d520fcadf9efaddeb3b1ee4"
Oct 29 23:48:15.664303 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:15.664264    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"]
Oct 29 23:48:15.669545 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:15.669516    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7aeaad-85b8dcddc8-zz84f"]
Oct 29 23:48:15.887625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:15.887591    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:15.907992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:15.907957    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58" podUID="e990be60-57d3-49f5-961a-d40381905773" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:48:16.034184 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:16.034142    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:16.170905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:16.168303    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:16.726445 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:16.726408    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:16.781497 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:16.781459    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf20849b-2d40-42ca-82fa-a91198bb012b" path="/var/lib/kubelet/pods/cf20849b-2d40-42ca-82fa-a91198bb012b/volumes"
Oct 29 23:48:16.857015 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:16.856982    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:16.915541 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:16.915504    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29" podUID="103e3adb-1b76-44e2-ba11-03d11fea6b70" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:48:17.044732 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:17.044651    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:17.181252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:17.181220    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:17.367711 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:48:17.367675    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:48:17.517440 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:17.517409    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:18.885920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:18.885890    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:20.732317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:20.732285    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:22.171074 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:22.171044    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:22.316010 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:22.315956    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:22.617347 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:22.617312    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:23.256278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:23.256238    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:23.379811 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:23.379760    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:23.631956 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:23.631915    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:24.670636 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:24.670602    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:24.805924 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:24.805890    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:25.019509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:25.019425    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"]
Oct 29 23:48:25.020038 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:25.020009    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264" podUID="f56ff730-ff0f-47f0-be77-3bbde2f85301" containerName="istio-proxy" containerID="cri-o://210dd2597dd76da398bc428e7f16635128420ce3c233f35db49009864f96737f" gracePeriod=30
Oct 29 23:48:25.038141 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:25.038112    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b"]
Oct 29 23:48:25.056587 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:25.056558    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:25.177243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:25.177215    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:25.321910 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:25.319203    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:25.460784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:25.460748    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:25.908402 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:25.908359    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58" podUID="e990be60-57d3-49f5-961a-d40381905773" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:48:25.926134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:25.926099    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:26.915411 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:26.915370    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29" podUID="103e3adb-1b76-44e2-ba11-03d11fea6b70" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:48:27.192114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:27.191998    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:27.775846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:27.775796    5428 generic.go:356] "Generic (PLEG): container finished" podID="103e3adb-1b76-44e2-ba11-03d11fea6b70" containerID="ffe14db32e335551daa2be51aec2bcd36e4f081028c37b0e9b0db0a7aff3eb00" exitCode=0
Oct 29 23:48:27.775972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:27.775860    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29" event={"ID":"103e3adb-1b76-44e2-ba11-03d11fea6b70","Type":"ContainerDied","Data":"ffe14db32e335551daa2be51aec2bcd36e4f081028c37b0e9b0db0a7aff3eb00"}
Oct 29 23:48:27.780657 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:27.780628    5428 patch_prober.go:29] interesting pod/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:48:27.780786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:27.780685    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264" podUID="f56ff730-ff0f-47f0-be77-3bbde2f85301" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:48:28.003095 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.003028    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29"
Oct 29 23:48:28.022950 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.022911    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103e3adb-1b76-44e2-ba11-03d11fea6b70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103e3adb-1b76-44e2-ba11-03d11fea6b70-plugins-config-volume\") pod \"103e3adb-1b76-44e2-ba11-03d11fea6b70\" (UID: \"103e3adb-1b76-44e2-ba11-03d11fea6b70\") "
Oct 29 23:48:28.023542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.023516    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103e3adb-1b76-44e2-ba11-03d11fea6b70-kube-api-access-qkp2b\" (UniqueName: \"kubernetes.io/projected/103e3adb-1b76-44e2-ba11-03d11fea6b70-kube-api-access-qkp2b\") pod \"103e3adb-1b76-44e2-ba11-03d11fea6b70\" (UID: \"103e3adb-1b76-44e2-ba11-03d11fea6b70\") "
Oct 29 23:48:28.023761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.023312    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103e3adb-1b76-44e2-ba11-03d11fea6b70-plugins-config-volume" pod "103e3adb-1b76-44e2-ba11-03d11fea6b70" (UID: "103e3adb-1b76-44e2-ba11-03d11fea6b70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:48:28.024481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.024457    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103e3adb-1b76-44e2-ba11-03d11fea6b70-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:28.034406 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.034373    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103e3adb-1b76-44e2-ba11-03d11fea6b70-kube-api-access-qkp2b" pod "103e3adb-1b76-44e2-ba11-03d11fea6b70" (UID: "103e3adb-1b76-44e2-ba11-03d11fea6b70"). InnerVolumeSpecName "kube-api-access-qkp2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:48:28.037428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.037372    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:28.100850 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.100795    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58"
Oct 29 23:48:28.125179 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.125155    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e990be60-57d3-49f5-961a-d40381905773-kube-api-access-42hqp\" (UniqueName: \"kubernetes.io/projected/e990be60-57d3-49f5-961a-d40381905773-kube-api-access-42hqp\") pod \"e990be60-57d3-49f5-961a-d40381905773\" (UID: \"e990be60-57d3-49f5-961a-d40381905773\") "
Oct 29 23:48:28.125391 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.125366    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e990be60-57d3-49f5-961a-d40381905773-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e990be60-57d3-49f5-961a-d40381905773-plugins-config-volume\") pod \"e990be60-57d3-49f5-961a-d40381905773\" (UID: \"e990be60-57d3-49f5-961a-d40381905773\") "
Oct 29 23:48:28.125725 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.125701    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e990be60-57d3-49f5-961a-d40381905773-plugins-config-volume" pod "e990be60-57d3-49f5-961a-d40381905773" (UID: "e990be60-57d3-49f5-961a-d40381905773"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:48:28.126067 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.126045    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e990be60-57d3-49f5-961a-d40381905773-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:28.126148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.126070    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkp2b\" (UniqueName: \"kubernetes.io/projected/103e3adb-1b76-44e2-ba11-03d11fea6b70-kube-api-access-qkp2b\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:28.127372 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.127347    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e990be60-57d3-49f5-961a-d40381905773-kube-api-access-42hqp" pod "e990be60-57d3-49f5-961a-d40381905773" (UID: "e990be60-57d3-49f5-961a-d40381905773"). InnerVolumeSpecName "kube-api-access-42hqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:48:28.184386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.184345    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:28.227363 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.227324    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42hqp\" (UniqueName: \"kubernetes.io/projected/e990be60-57d3-49f5-961a-d40381905773-kube-api-access-42hqp\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:28.313984 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.313903    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:28.719605 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.719572    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:28.784980 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.784949    5428 generic.go:356] "Generic (PLEG): container finished" podID="e990be60-57d3-49f5-961a-d40381905773" containerID="33238e783b51fee2b7ffb3496f342d8142d6b458fd743b2fc802e337710e0003" exitCode=0
Oct 29 23:48:28.785155 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.784988    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58"
Oct 29 23:48:28.785155 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.785032    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58" event={"ID":"e990be60-57d3-49f5-961a-d40381905773","Type":"ContainerDied","Data":"33238e783b51fee2b7ffb3496f342d8142d6b458fd743b2fc802e337710e0003"}
Oct 29 23:48:28.785155 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.785088    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58" event={"ID":"e990be60-57d3-49f5-961a-d40381905773","Type":"ContainerDied","Data":"190304d858967328457aeefc529135244a345da3ea7fc1bef2b515efc355ded8"}
Oct 29 23:48:28.785155 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.785123    5428 scope.go:122] "RemoveContainer" containerID="33238e783b51fee2b7ffb3496f342d8142d6b458fd743b2fc802e337710e0003"
Oct 29 23:48:28.787047 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.787024    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29" event={"ID":"103e3adb-1b76-44e2-ba11-03d11fea6b70","Type":"ContainerDied","Data":"f4f0010911b5ada9dd4ddbc6b6579cffa3fd85946a450515ee16c088e37799d5"}
Oct 29 23:48:28.787141 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.787107    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29"
Oct 29 23:48:28.801538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.801523    5428 scope.go:122] "RemoveContainer" containerID="33238e783b51fee2b7ffb3496f342d8142d6b458fd743b2fc802e337710e0003"
Oct 29 23:48:28.801844 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:48:28.801821    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33238e783b51fee2b7ffb3496f342d8142d6b458fd743b2fc802e337710e0003\": container with ID starting with 33238e783b51fee2b7ffb3496f342d8142d6b458fd743b2fc802e337710e0003 not found: ID does not exist" containerID="33238e783b51fee2b7ffb3496f342d8142d6b458fd743b2fc802e337710e0003"
Oct 29 23:48:28.801904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.801854    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33238e783b51fee2b7ffb3496f342d8142d6b458fd743b2fc802e337710e0003"} err="failed to get container status \"33238e783b51fee2b7ffb3496f342d8142d6b458fd743b2fc802e337710e0003\": rpc error: code = NotFound desc = could not find container \"33238e783b51fee2b7ffb3496f342d8142d6b458fd743b2fc802e337710e0003\": container with ID starting with 33238e783b51fee2b7ffb3496f342d8142d6b458fd743b2fc802e337710e0003 not found: ID does not exist"
Oct 29 23:48:28.801904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.801869    5428 scope.go:122] "RemoveContainer" containerID="ffe14db32e335551daa2be51aec2bcd36e4f081028c37b0e9b0db0a7aff3eb00"
Oct 29 23:48:28.847810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.847775    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58"]
Oct 29 23:48:28.861707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.861673    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-26a2a4-54557bf459-zlc58"]
Oct 29 23:48:28.887954 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.887924    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29"]
Oct 29 23:48:28.892311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:28.892286    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-26a2a4-7dcbf98fbd-qsn29"]
Oct 29 23:48:30.038409 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.038370    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b" podUID="526c46bd-79b3-446f-8b63-7c341d98b102" containerName="payload-pre-processing" containerID="cri-o://6d109caddac23d6fae3e37515aca77940ce1f4e1980ed70a463c741b99940a02" gracePeriod=25
Oct 29 23:48:30.181723 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.181694    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:30.321310 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.321288    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 23:48:30.355075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.355047    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-workload-certs\") pod \"f56ff730-ff0f-47f0-be77-3bbde2f85301\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") "
Oct 29 23:48:30.355263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.355103    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-workload-socket\") pod \"f56ff730-ff0f-47f0-be77-3bbde2f85301\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") "
Oct 29 23:48:30.355263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.355163    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f56ff730-ff0f-47f0-be77-3bbde2f85301-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f56ff730-ff0f-47f0-be77-3bbde2f85301-istiod-ca-cert\") pod \"f56ff730-ff0f-47f0-be77-3bbde2f85301\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") "
Oct 29 23:48:30.355263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.355213    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f56ff730-ff0f-47f0-be77-3bbde2f85301-kube-api-access-z9j2b\" (UniqueName: \"kubernetes.io/projected/f56ff730-ff0f-47f0-be77-3bbde2f85301-kube-api-access-z9j2b\") pod \"f56ff730-ff0f-47f0-be77-3bbde2f85301\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") "
Oct 29 23:48:30.355263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.355244    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-data\") pod \"f56ff730-ff0f-47f0-be77-3bbde2f85301\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") "
Oct 29 23:48:30.355500 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.355302    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-podinfo\") pod \"f56ff730-ff0f-47f0-be77-3bbde2f85301\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") "
Oct 29 23:48:30.355500 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.355356    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-workload-certs" pod "f56ff730-ff0f-47f0-be77-3bbde2f85301" (UID: "f56ff730-ff0f-47f0-be77-3bbde2f85301"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:48:30.355500 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.355417    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-envoy\") pod \"f56ff730-ff0f-47f0-be77-3bbde2f85301\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") "
Oct 29 23:48:30.355500 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.355454    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-token\" (UniqueName: \"kubernetes.io/projected/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-token\") pod \"f56ff730-ff0f-47f0-be77-3bbde2f85301\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") "
Oct 29 23:48:30.355500 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.355460    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-workload-socket" pod "f56ff730-ff0f-47f0-be77-3bbde2f85301" (UID: "f56ff730-ff0f-47f0-be77-3bbde2f85301"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:48:30.355790 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.355507    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-credential-socket\") pod \"f56ff730-ff0f-47f0-be77-3bbde2f85301\" (UID: \"f56ff730-ff0f-47f0-be77-3bbde2f85301\") "
Oct 29 23:48:30.355790 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.355519    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-data" pod "f56ff730-ff0f-47f0-be77-3bbde2f85301" (UID: "f56ff730-ff0f-47f0-be77-3bbde2f85301"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:48:30.355790 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.355622    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f56ff730-ff0f-47f0-be77-3bbde2f85301-istiod-ca-cert" pod "f56ff730-ff0f-47f0-be77-3bbde2f85301" (UID: "f56ff730-ff0f-47f0-be77-3bbde2f85301"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:48:30.356072 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.356034    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-credential-socket" pod "f56ff730-ff0f-47f0-be77-3bbde2f85301" (UID: "f56ff730-ff0f-47f0-be77-3bbde2f85301"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:48:30.356270 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.356249    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:30.356356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.356277    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:30.356356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.356292    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:30.356356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.356308    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f56ff730-ff0f-47f0-be77-3bbde2f85301-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:30.356356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.356321    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:30.358178 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.358151    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56ff730-ff0f-47f0-be77-3bbde2f85301-kube-api-access-z9j2b" pod "f56ff730-ff0f-47f0-be77-3bbde2f85301" (UID: "f56ff730-ff0f-47f0-be77-3bbde2f85301"). InnerVolumeSpecName "kube-api-access-z9j2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:48:30.358343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.358324    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-podinfo" pod "f56ff730-ff0f-47f0-be77-3bbde2f85301" (UID: "f56ff730-ff0f-47f0-be77-3bbde2f85301"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:48:30.358429 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.358376    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-envoy" pod "f56ff730-ff0f-47f0-be77-3bbde2f85301" (UID: "f56ff730-ff0f-47f0-be77-3bbde2f85301"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:48:30.358779 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.358759    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-token" pod "f56ff730-ff0f-47f0-be77-3bbde2f85301" (UID: "f56ff730-ff0f-47f0-be77-3bbde2f85301"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:48:30.458058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.458022    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:30.458058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.458047    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:30.458058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.458057    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f56ff730-ff0f-47f0-be77-3bbde2f85301-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:30.458058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.458066    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9j2b\" (UniqueName: \"kubernetes.io/projected/f56ff730-ff0f-47f0-be77-3bbde2f85301-kube-api-access-z9j2b\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:30.655527 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.655490    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b" podUID="526c46bd-79b3-446f-8b63-7c341d98b102" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:48:30.778832 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.778785    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103e3adb-1b76-44e2-ba11-03d11fea6b70" path="/var/lib/kubelet/pods/103e3adb-1b76-44e2-ba11-03d11fea6b70/volumes"
Oct 29 23:48:30.779286 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.779272    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e990be60-57d3-49f5-961a-d40381905773" path="/var/lib/kubelet/pods/e990be60-57d3-49f5-961a-d40381905773/volumes"
Oct 29 23:48:30.805203 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.805174    5428 generic.go:356] "Generic (PLEG): container finished" podID="f56ff730-ff0f-47f0-be77-3bbde2f85301" containerID="210dd2597dd76da398bc428e7f16635128420ce3c233f35db49009864f96737f" exitCode=0
Oct 29 23:48:30.805342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.805220    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"
Oct 29 23:48:30.805342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.805255    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264" event={"ID":"f56ff730-ff0f-47f0-be77-3bbde2f85301","Type":"ContainerDied","Data":"210dd2597dd76da398bc428e7f16635128420ce3c233f35db49009864f96737f"}
Oct 29 23:48:30.805342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.805320    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264" event={"ID":"f56ff730-ff0f-47f0-be77-3bbde2f85301","Type":"ContainerDied","Data":"1a240118edb2b76554337af641d90f4dbd7a4ca351fd375e39d57701798988c2"}
Oct 29 23:48:30.805505 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.805347    5428 scope.go:122] "RemoveContainer" containerID="210dd2597dd76da398bc428e7f16635128420ce3c233f35db49009864f96737f"
Oct 29 23:48:30.819518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.819500    5428 scope.go:122] "RemoveContainer" containerID="210dd2597dd76da398bc428e7f16635128420ce3c233f35db49009864f96737f"
Oct 29 23:48:30.819977 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:48:30.819943    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"210dd2597dd76da398bc428e7f16635128420ce3c233f35db49009864f96737f\": container with ID starting with 210dd2597dd76da398bc428e7f16635128420ce3c233f35db49009864f96737f not found: ID does not exist" containerID="210dd2597dd76da398bc428e7f16635128420ce3c233f35db49009864f96737f"
Oct 29 23:48:30.820088 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.819988    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"210dd2597dd76da398bc428e7f16635128420ce3c233f35db49009864f96737f"} err="failed to get container status \"210dd2597dd76da398bc428e7f16635128420ce3c233f35db49009864f96737f\": rpc error: code = NotFound desc = could not find container \"210dd2597dd76da398bc428e7f16635128420ce3c233f35db49009864f96737f\": container with ID starting with 210dd2597dd76da398bc428e7f16635128420ce3c233f35db49009864f96737f not found: ID does not exist"
Oct 29 23:48:30.859370 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.859337    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"]
Oct 29 23:48:30.872109 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:30.872082    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7aeaad-openshift-default-7fdbbc697f-jg264"]
Oct 29 23:48:31.976034 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:31.975999    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:32.778255 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:32.778206    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f56ff730-ff0f-47f0-be77-3bbde2f85301" path="/var/lib/kubelet/pods/f56ff730-ff0f-47f0-be77-3bbde2f85301/volumes"
Oct 29 23:48:33.468481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:33.468448    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:33.806958 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:33.806877    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:34.265218 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:34.265183    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:34.856022 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:34.855968    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:34.979951 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:34.979919    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:35.554102 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:35.554072    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:35.683762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:35.683728    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:35.962991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:35.962957    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:36.781087 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:36.781055    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k"]
Oct 29 23:48:36.798739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:36.798710    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx"]
Oct 29 23:48:36.908573 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:36.908541    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:37.037548 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:37.037439    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:38.005021 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:38.004989    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:39.425762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.425730    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"]
Oct 29 23:48:39.426242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.425955    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb" podUID="980d539c-9fe0-43d8-bff8-fcdb9aa963d9" containerName="maas-api" containerID="cri-o://4326986f2e356b8285a2fe992546e1886eb1297f5d7be1039caf041090d44214" gracePeriod=30
Oct 29 23:48:39.507017 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.506992    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:39.728753 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.728405    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"
Oct 29 23:48:39.798224 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.798186    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-maas-api-tls\") pod \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\" (UID: \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\") "
Oct 29 23:48:39.798434 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.798288    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-kube-api-access-htj79\" (UniqueName: \"kubernetes.io/projected/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-kube-api-access-htj79\") pod \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\" (UID: \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\") "
Oct 29 23:48:39.798434 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.798400    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-metrics-certs\") pod \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\" (UID: \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\") "
Oct 29 23:48:39.798581 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.798503    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-openshift-service-ca\") pod \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\" (UID: \"980d539c-9fe0-43d8-bff8-fcdb9aa963d9\") "
Oct 29 23:48:39.798971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.798939    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-openshift-service-ca" pod "980d539c-9fe0-43d8-bff8-fcdb9aa963d9" (UID: "980d539c-9fe0-43d8-bff8-fcdb9aa963d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:48:39.799436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.799408    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:39.800654 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.800624    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-maas-api-tls" pod "980d539c-9fe0-43d8-bff8-fcdb9aa963d9" (UID: "980d539c-9fe0-43d8-bff8-fcdb9aa963d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:48:39.801011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.800984    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-kube-api-access-htj79" pod "980d539c-9fe0-43d8-bff8-fcdb9aa963d9" (UID: "980d539c-9fe0-43d8-bff8-fcdb9aa963d9"). InnerVolumeSpecName "kube-api-access-htj79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:48:39.801277 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.801251    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-metrics-certs" pod "980d539c-9fe0-43d8-bff8-fcdb9aa963d9" (UID: "980d539c-9fe0-43d8-bff8-fcdb9aa963d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:48:39.900159 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.900116    5428 generic.go:356] "Generic (PLEG): container finished" podID="980d539c-9fe0-43d8-bff8-fcdb9aa963d9" containerID="4326986f2e356b8285a2fe992546e1886eb1297f5d7be1039caf041090d44214" exitCode=0
Oct 29 23:48:39.900362 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.900166    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"
Oct 29 23:48:39.900362 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.900182    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb" event={"ID":"980d539c-9fe0-43d8-bff8-fcdb9aa963d9","Type":"ContainerDied","Data":"4326986f2e356b8285a2fe992546e1886eb1297f5d7be1039caf041090d44214"}
Oct 29 23:48:39.900362 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.900251    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb" event={"ID":"980d539c-9fe0-43d8-bff8-fcdb9aa963d9","Type":"ContainerDied","Data":"ed9e5f0a274b2f3c19093a262071e2a398fe13b1e64efb37c4ace48be550fdb7"}
Oct 29 23:48:39.900362 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.900301    5428 scope.go:122] "RemoveContainer" containerID="4326986f2e356b8285a2fe992546e1886eb1297f5d7be1039caf041090d44214"
Oct 29 23:48:39.901962 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.901933    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htj79\" (UniqueName: \"kubernetes.io/projected/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-kube-api-access-htj79\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:39.901962 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.901965    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:39.902213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.901980    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/980d539c-9fe0-43d8-bff8-fcdb9aa963d9-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:39.991148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.989032    5428 scope.go:122] "RemoveContainer" containerID="4326986f2e356b8285a2fe992546e1886eb1297f5d7be1039caf041090d44214"
Oct 29 23:48:39.991148 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:48:39.989412    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4326986f2e356b8285a2fe992546e1886eb1297f5d7be1039caf041090d44214\": container with ID starting with 4326986f2e356b8285a2fe992546e1886eb1297f5d7be1039caf041090d44214 not found: ID does not exist" containerID="4326986f2e356b8285a2fe992546e1886eb1297f5d7be1039caf041090d44214"
Oct 29 23:48:39.991148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.989450    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4326986f2e356b8285a2fe992546e1886eb1297f5d7be1039caf041090d44214"} err="failed to get container status \"4326986f2e356b8285a2fe992546e1886eb1297f5d7be1039caf041090d44214\": rpc error: code = NotFound desc = could not find container \"4326986f2e356b8285a2fe992546e1886eb1297f5d7be1039caf041090d44214\": container with ID starting with 4326986f2e356b8285a2fe992546e1886eb1297f5d7be1039caf041090d44214 not found: ID does not exist"
Oct 29 23:48:39.998852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:39.997509    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"]
Oct 29 23:48:40.043521 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:40.043490    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4ad17e-64ccdc788c-9cmfb"]
Oct 29 23:48:40.303894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:40.303779    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:40.420610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:40.420582    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:40.560796 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:40.560710    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:40.655707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:40.655661    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b" podUID="526c46bd-79b3-446f-8b63-7c341d98b102" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:48:40.778482 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:40.778447    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="980d539c-9fe0-43d8-bff8-fcdb9aa963d9" path="/var/lib/kubelet/pods/980d539c-9fe0-43d8-bff8-fcdb9aa963d9/volumes"
Oct 29 23:48:40.923908 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:40.923873    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:41.782028 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:41.781976    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k" podUID="3d79a8de-da14-43a3-b4f1-cdb7fd66d235" containerName="payload-processing" containerID="cri-o://0fa64e86b0825984e8f80f7380e676982bac1b1884556e341cc89797ae0f7c26" gracePeriod=25
Oct 29 23:48:41.799664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:41.799622    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx" podUID="4b8035b7-d41e-43bf-842e-9f9f9e54eae9" containerName="payload-pre-processing" containerID="cri-o://d2da6df1945e029e8e3f20c6a337426d63bfdffa9d818025b7b0a0ae8239bf38" gracePeriod=25
Oct 29 23:48:41.920433 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:41.920398    5428 generic.go:356] "Generic (PLEG): container finished" podID="4b8035b7-d41e-43bf-842e-9f9f9e54eae9" containerID="d2da6df1945e029e8e3f20c6a337426d63bfdffa9d818025b7b0a0ae8239bf38" exitCode=0
Oct 29 23:48:41.920561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:41.920470    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx" event={"ID":"4b8035b7-d41e-43bf-842e-9f9f9e54eae9","Type":"ContainerDied","Data":"d2da6df1945e029e8e3f20c6a337426d63bfdffa9d818025b7b0a0ae8239bf38"}
Oct 29 23:48:41.922697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:41.922672    5428 generic.go:356] "Generic (PLEG): container finished" podID="3d79a8de-da14-43a3-b4f1-cdb7fd66d235" containerID="0fa64e86b0825984e8f80f7380e676982bac1b1884556e341cc89797ae0f7c26" exitCode=0
Oct 29 23:48:41.922823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:41.922749    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k" event={"ID":"3d79a8de-da14-43a3-b4f1-cdb7fd66d235","Type":"ContainerDied","Data":"0fa64e86b0825984e8f80f7380e676982bac1b1884556e341cc89797ae0f7c26"}
Oct 29 23:48:42.095156 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.095127    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k"
Oct 29 23:48:42.131261 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.131237    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx"
Oct 29 23:48:42.250708 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.250670    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b8035b7-d41e-43bf-842e-9f9f9e54eae9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8035b7-d41e-43bf-842e-9f9f9e54eae9-plugins-config-volume\") pod \"4b8035b7-d41e-43bf-842e-9f9f9e54eae9\" (UID: \"4b8035b7-d41e-43bf-842e-9f9f9e54eae9\") "
Oct 29 23:48:42.250916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.250718    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d79a8de-da14-43a3-b4f1-cdb7fd66d235-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d79a8de-da14-43a3-b4f1-cdb7fd66d235-plugins-config-volume\") pod \"3d79a8de-da14-43a3-b4f1-cdb7fd66d235\" (UID: \"3d79a8de-da14-43a3-b4f1-cdb7fd66d235\") "
Oct 29 23:48:42.250916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.250863    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d79a8de-da14-43a3-b4f1-cdb7fd66d235-kube-api-access-pp76h\" (UniqueName: \"kubernetes.io/projected/3d79a8de-da14-43a3-b4f1-cdb7fd66d235-kube-api-access-pp76h\") pod \"3d79a8de-da14-43a3-b4f1-cdb7fd66d235\" (UID: \"3d79a8de-da14-43a3-b4f1-cdb7fd66d235\") "
Oct 29 23:48:42.251078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.251010    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8035b7-d41e-43bf-842e-9f9f9e54eae9-kube-api-access-xvwtx\" (UniqueName: \"kubernetes.io/projected/4b8035b7-d41e-43bf-842e-9f9f9e54eae9-kube-api-access-xvwtx\") pod \"4b8035b7-d41e-43bf-842e-9f9f9e54eae9\" (UID: \"4b8035b7-d41e-43bf-842e-9f9f9e54eae9\") "
Oct 29 23:48:42.251078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.251059    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8035b7-d41e-43bf-842e-9f9f9e54eae9-plugins-config-volume" pod "4b8035b7-d41e-43bf-842e-9f9f9e54eae9" (UID: "4b8035b7-d41e-43bf-842e-9f9f9e54eae9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:48:42.251203 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.251174    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d79a8de-da14-43a3-b4f1-cdb7fd66d235-plugins-config-volume" pod "3d79a8de-da14-43a3-b4f1-cdb7fd66d235" (UID: "3d79a8de-da14-43a3-b4f1-cdb7fd66d235"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:48:42.251649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.251628    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8035b7-d41e-43bf-842e-9f9f9e54eae9-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:42.251649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.251648    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d79a8de-da14-43a3-b4f1-cdb7fd66d235-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:42.253083 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.253062    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d79a8de-da14-43a3-b4f1-cdb7fd66d235-kube-api-access-pp76h" pod "3d79a8de-da14-43a3-b4f1-cdb7fd66d235" (UID: "3d79a8de-da14-43a3-b4f1-cdb7fd66d235"). InnerVolumeSpecName "kube-api-access-pp76h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:48:42.253179 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.253138    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8035b7-d41e-43bf-842e-9f9f9e54eae9-kube-api-access-xvwtx" pod "4b8035b7-d41e-43bf-842e-9f9f9e54eae9" (UID: "4b8035b7-d41e-43bf-842e-9f9f9e54eae9"). InnerVolumeSpecName "kube-api-access-xvwtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:48:42.352899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.352824    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp76h\" (UniqueName: \"kubernetes.io/projected/3d79a8de-da14-43a3-b4f1-cdb7fd66d235-kube-api-access-pp76h\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:42.352899 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.352849    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvwtx\" (UniqueName: \"kubernetes.io/projected/4b8035b7-d41e-43bf-842e-9f9f9e54eae9-kube-api-access-xvwtx\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:42.413992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.413960    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:42.931517 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.931482    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx"
Oct 29 23:48:42.931978 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.931489    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx" event={"ID":"4b8035b7-d41e-43bf-842e-9f9f9e54eae9","Type":"ContainerDied","Data":"c95befa41d8bfa82f2f603a963449d767d48854d9d5eaa835ede6f08efb00bc6"}
Oct 29 23:48:42.931978 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.931645    5428 scope.go:122] "RemoveContainer" containerID="d2da6df1945e029e8e3f20c6a337426d63bfdffa9d818025b7b0a0ae8239bf38"
Oct 29 23:48:42.933297 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.933278    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k"
Oct 29 23:48:42.933415 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.933299    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k" event={"ID":"3d79a8de-da14-43a3-b4f1-cdb7fd66d235","Type":"ContainerDied","Data":"8d3429acc28b9d864bc0fc8393258390550d72fb02a4113c4e645c2edbb850b4"}
Oct 29 23:48:42.943134 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.943119    5428 scope.go:122] "RemoveContainer" containerID="0fa64e86b0825984e8f80f7380e676982bac1b1884556e341cc89797ae0f7c26"
Oct 29 23:48:42.987006 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.986981    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k"]
Oct 29 23:48:42.999784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:42.999758    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4ad17e-6f76d898cd-fnq4k"]
Oct 29 23:48:43.036749 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:43.036720    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx"]
Oct 29 23:48:43.045148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:43.045085    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4ad17e-67f8c95d7-fr6mx"]
Oct 29 23:48:44.188486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:44.188455    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:44.779343 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:44.779309    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d79a8de-da14-43a3-b4f1-cdb7fd66d235" path="/var/lib/kubelet/pods/3d79a8de-da14-43a3-b4f1-cdb7fd66d235/volumes"
Oct 29 23:48:44.779795 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:44.779781    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b8035b7-d41e-43bf-842e-9f9f9e54eae9" path="/var/lib/kubelet/pods/4b8035b7-d41e-43bf-842e-9f9f9e54eae9/volumes"
Oct 29 23:48:45.895936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:45.895903    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:46.028286 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:46.028255    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:46.422133 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:46.422104    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:47.368778 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:48:47.368722    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:48:47.526846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:47.522387    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:47.660944 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:47.660864    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:47.783832 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:47.781936    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"]
Oct 29 23:48:47.783832 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:47.782149    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm" podUID="45b6bf03-44e3-43c8-9ed9-7984e56e031e" containerName="istio-proxy" containerID="cri-o://f6aa11558873bdedd9ea069b9522067fe4ef1dbd0a60c626b5b07ae72190d746" gracePeriod=30
Oct 29 23:48:47.836521 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:47.836488    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:48.241505 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:48.241472    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:48.417039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:48.417003    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:50.332036 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:50.332007    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b"
Oct 29 23:48:50.341201 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:50.341174    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:50.385862 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:50.385829    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/526c46bd-79b3-446f-8b63-7c341d98b102-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526c46bd-79b3-446f-8b63-7c341d98b102-plugins-config-volume\") pod \"526c46bd-79b3-446f-8b63-7c341d98b102\" (UID: \"526c46bd-79b3-446f-8b63-7c341d98b102\") "
Oct 29 23:48:50.386069 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:50.385911    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/526c46bd-79b3-446f-8b63-7c341d98b102-kube-api-access-fmr7b\" (UniqueName: \"kubernetes.io/projected/526c46bd-79b3-446f-8b63-7c341d98b102-kube-api-access-fmr7b\") pod \"526c46bd-79b3-446f-8b63-7c341d98b102\" (UID: \"526c46bd-79b3-446f-8b63-7c341d98b102\") "
Oct 29 23:48:50.386331 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:50.386303    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/526c46bd-79b3-446f-8b63-7c341d98b102-plugins-config-volume" pod "526c46bd-79b3-446f-8b63-7c341d98b102" (UID: "526c46bd-79b3-446f-8b63-7c341d98b102"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:48:50.386760 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:50.386718    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526c46bd-79b3-446f-8b63-7c341d98b102-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:50.388524 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:50.388498    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/526c46bd-79b3-446f-8b63-7c341d98b102-kube-api-access-fmr7b" pod "526c46bd-79b3-446f-8b63-7c341d98b102" (UID: "526c46bd-79b3-446f-8b63-7c341d98b102"). InnerVolumeSpecName "kube-api-access-fmr7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:48:50.488081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:50.488052    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmr7b\" (UniqueName: \"kubernetes.io/projected/526c46bd-79b3-446f-8b63-7c341d98b102-kube-api-access-fmr7b\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:51.012313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:51.012282    5428 generic.go:356] "Generic (PLEG): container finished" podID="526c46bd-79b3-446f-8b63-7c341d98b102" containerID="6d109caddac23d6fae3e37515aca77940ce1f4e1980ed70a463c741b99940a02" exitCode=0
Oct 29 23:48:51.012525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:51.012329    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b"
Oct 29 23:48:51.012525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:51.012372    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b" event={"ID":"526c46bd-79b3-446f-8b63-7c341d98b102","Type":"ContainerDied","Data":"6d109caddac23d6fae3e37515aca77940ce1f4e1980ed70a463c741b99940a02"}
Oct 29 23:48:51.012525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:51.012430    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b" event={"ID":"526c46bd-79b3-446f-8b63-7c341d98b102","Type":"ContainerDied","Data":"a184754d13eda23995fc614ebbc3ad9783bb3a27c19b1d5d691765e3a0d40f33"}
Oct 29 23:48:51.012525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:51.012458    5428 scope.go:122] "RemoveContainer" containerID="6d109caddac23d6fae3e37515aca77940ce1f4e1980ed70a463c741b99940a02"
Oct 29 23:48:51.023756 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:51.023738    5428 scope.go:122] "RemoveContainer" containerID="6d109caddac23d6fae3e37515aca77940ce1f4e1980ed70a463c741b99940a02"
Oct 29 23:48:51.024061 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:48:51.024040    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d109caddac23d6fae3e37515aca77940ce1f4e1980ed70a463c741b99940a02\": container with ID starting with 6d109caddac23d6fae3e37515aca77940ce1f4e1980ed70a463c741b99940a02 not found: ID does not exist" containerID="6d109caddac23d6fae3e37515aca77940ce1f4e1980ed70a463c741b99940a02"
Oct 29 23:48:51.024116 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:51.024070    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d109caddac23d6fae3e37515aca77940ce1f4e1980ed70a463c741b99940a02"} err="failed to get container status \"6d109caddac23d6fae3e37515aca77940ce1f4e1980ed70a463c741b99940a02\": rpc error: code = NotFound desc = could not find container \"6d109caddac23d6fae3e37515aca77940ce1f4e1980ed70a463c741b99940a02\": container with ID starting with 6d109caddac23d6fae3e37515aca77940ce1f4e1980ed70a463c741b99940a02 not found: ID does not exist"
Oct 29 23:48:51.068301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:51.068232    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b"]
Oct 29 23:48:51.080333 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:51.080309    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-52f67b-79bcd8c75-xpb7b"]
Oct 29 23:48:51.231473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:51.231434    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:48:51.231645 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:51.231512    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm" podUID="45b6bf03-44e3-43c8-9ed9-7984e56e031e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:48:51.627952 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:51.627917    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:52.410102 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:52.410064    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:52.780231 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:52.780157    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="526c46bd-79b3-446f-8b63-7c341d98b102" path="/var/lib/kubelet/pods/526c46bd-79b3-446f-8b63-7c341d98b102/volumes"
Oct 29 23:48:53.036932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.036585    5428 generic.go:356] "Generic (PLEG): container finished" podID="45b6bf03-44e3-43c8-9ed9-7984e56e031e" containerID="f6aa11558873bdedd9ea069b9522067fe4ef1dbd0a60c626b5b07ae72190d746" exitCode=0
Oct 29 23:48:53.036932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.036720    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm" event={"ID":"45b6bf03-44e3-43c8-9ed9-7984e56e031e","Type":"ContainerDied","Data":"f6aa11558873bdedd9ea069b9522067fe4ef1dbd0a60c626b5b07ae72190d746"}
Oct 29 23:48:53.091250 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.091212    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:48:53.217429 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.217394    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"]
Oct 29 23:48:53.227020 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.226932    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-workload-socket\") pod \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") "
Oct 29 23:48:53.227020 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.226987    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-token\" (UniqueName: \"kubernetes.io/projected/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-token\") pod \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") "
Oct 29 23:48:53.227020 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.227015    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-workload-certs\") pod \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") "
Oct 29 23:48:53.227390 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.227073    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-data\") pod \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") "
Oct 29 23:48:53.227390 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.227149    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-credential-socket\") pod \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") "
Oct 29 23:48:53.227390 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.227183    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-envoy\") pod \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") "
Oct 29 23:48:53.227390 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.227221    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45b6bf03-44e3-43c8-9ed9-7984e56e031e-kube-api-access-nf5g5\" (UniqueName: \"kubernetes.io/projected/45b6bf03-44e3-43c8-9ed9-7984e56e031e-kube-api-access-nf5g5\") pod \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") "
Oct 29 23:48:53.227390 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.227236    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-workload-socket" pod "45b6bf03-44e3-43c8-9ed9-7984e56e031e" (UID: "45b6bf03-44e3-43c8-9ed9-7984e56e031e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:48:53.227675 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.227571    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-data" pod "45b6bf03-44e3-43c8-9ed9-7984e56e031e" (UID: "45b6bf03-44e3-43c8-9ed9-7984e56e031e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:48:53.227756 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.227731    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-credential-socket" pod "45b6bf03-44e3-43c8-9ed9-7984e56e031e" (UID: "45b6bf03-44e3-43c8-9ed9-7984e56e031e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:48:53.227979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.227781    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-workload-certs" pod "45b6bf03-44e3-43c8-9ed9-7984e56e031e" (UID: "45b6bf03-44e3-43c8-9ed9-7984e56e031e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:48:53.228160 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.228137    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istiod-ca-cert" pod "45b6bf03-44e3-43c8-9ed9-7984e56e031e" (UID: "45b6bf03-44e3-43c8-9ed9-7984e56e031e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:48:53.228361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.227262    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istiod-ca-cert\") pod \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") "
Oct 29 23:48:53.228484 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.228455    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-podinfo\") pod \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\" (UID: \"45b6bf03-44e3-43c8-9ed9-7984e56e031e\") "
Oct 29 23:48:53.229594 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.229564    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:53.229594 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.229595    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:53.229770 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.229609    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:53.229770 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.229622    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:53.229770 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.229634    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:53.230635 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.230609    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b6bf03-44e3-43c8-9ed9-7984e56e031e-kube-api-access-nf5g5" pod "45b6bf03-44e3-43c8-9ed9-7984e56e031e" (UID: "45b6bf03-44e3-43c8-9ed9-7984e56e031e"). InnerVolumeSpecName "kube-api-access-nf5g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:48:53.230749 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.230653    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-envoy" pod "45b6bf03-44e3-43c8-9ed9-7984e56e031e" (UID: "45b6bf03-44e3-43c8-9ed9-7984e56e031e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:48:53.231666 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.231644    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-podinfo" pod "45b6bf03-44e3-43c8-9ed9-7984e56e031e" (UID: "45b6bf03-44e3-43c8-9ed9-7984e56e031e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:48:53.232502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.232480    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-token" pod "45b6bf03-44e3-43c8-9ed9-7984e56e031e" (UID: "45b6bf03-44e3-43c8-9ed9-7984e56e031e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:48:53.331123 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.331026    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:53.331123 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.331064    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:53.331123 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.331079    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf5g5\" (UniqueName: \"kubernetes.io/projected/45b6bf03-44e3-43c8-9ed9-7984e56e031e-kube-api-access-nf5g5\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:53.331123 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.331092    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45b6bf03-44e3-43c8-9ed9-7984e56e031e-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:48:53.362124 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:53.362089    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:54.046509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:54.046478    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"
Oct 29 23:48:54.047021 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:54.046479    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm" event={"ID":"45b6bf03-44e3-43c8-9ed9-7984e56e031e","Type":"ContainerDied","Data":"fbbab6c96feea0ce6aeb0006c87d24fbd470348c2f2feb26c6f5e1c16782050c"}
Oct 29 23:48:54.047021 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:54.046625    5428 scope.go:122] "RemoveContainer" containerID="f6aa11558873bdedd9ea069b9522067fe4ef1dbd0a60c626b5b07ae72190d746"
Oct 29 23:48:54.114819 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:48:54.114778    5428 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b6bf03_44e3_43c8_9ed9_7984e56e031e.slice/crio-fbbab6c96feea0ce6aeb0006c87d24fbd470348c2f2feb26c6f5e1c16782050c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b6bf03_44e3_43c8_9ed9_7984e56e031e.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 23:48:54.116997 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:54.116919    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"]
Oct 29 23:48:54.131488 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:54.131459    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4ad17e-openshift-default-6976d9c869-pgscm"]
Oct 29 23:48:54.493065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:54.493024    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:54.778538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:54.778458    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45b6bf03-44e3-43c8-9ed9-7984e56e031e" path="/var/lib/kubelet/pods/45b6bf03-44e3-43c8-9ed9-7984e56e031e/volumes"
Oct 29 23:48:56.330770 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:56.330717    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:58.675622 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:58.675582    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:58.926268 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:58.926169    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7"]
Oct 29 23:48:58.973954 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:58.973918    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"]
Oct 29 23:48:59.116307 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:59.116275    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:48:59.245554 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:48:59.245459    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:00.870490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:00.870456    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:02.835167 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:02.835135    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:03.935318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:03.935264    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7" podUID="6f08d29e-2d26-454f-9667-8c3b99f6855d" containerName="payload-processing" containerID="cri-o://c6827f0beee74dd6d3e013e2db3b060391f2f3cad34a2c6d366a1ee5fb5cf5f0" gracePeriod=25
Oct 29 23:49:03.974260 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:03.974201    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff" podUID="5180d7a3-b01d-4639-9421-f802b9155fa2" containerName="payload-pre-processing" containerID="cri-o://69b36197811fa0e3274a486b874a3a5fddd57f70425cacedb8ad2ca0f04b7906" gracePeriod=25
Oct 29 23:49:04.283842 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:04.283718    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff" podUID="5180d7a3-b01d-4639-9421-f802b9155fa2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:49:04.283842 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:04.283717    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7" podUID="6f08d29e-2d26-454f-9667-8c3b99f6855d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:49:04.859072 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:04.859036    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:06.676364 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:06.676328    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:07.116218 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:07.116184    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs"]
Oct 29 23:49:07.225419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:07.225379    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:07.385893 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:07.385811    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:08.241091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.241051    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" containerID="cri-o://9ce48f686f5b438e9ee6617c2ec850607c5b0eecb59f9e4c3876e8770d1dd43e" gracePeriod=45
Oct 29 23:49:08.258773 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.258741    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": dial tcp 10.133.0.77:8000: connect: connection refused"
Oct 29 23:49:08.519394 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.519369    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 23:49:08.561324 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.561293    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-tmp-dir\") pod \"90966e36-5c24-4642-97a3-dbe4825b6a10\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") "
Oct 29 23:49:08.561476 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.561378    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-home\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-home\") pod \"90966e36-5c24-4642-97a3-dbe4825b6a10\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") "
Oct 29 23:49:08.561522 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.561492    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90966e36-5c24-4642-97a3-dbe4825b6a10-kube-api-access-qvpr4\" (UniqueName: \"kubernetes.io/projected/90966e36-5c24-4642-97a3-dbe4825b6a10-kube-api-access-qvpr4\") pod \"90966e36-5c24-4642-97a3-dbe4825b6a10\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") "
Oct 29 23:49:08.561580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.561554    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90966e36-5c24-4642-97a3-dbe4825b6a10-tls-certs\" (UniqueName: \"kubernetes.io/secret/90966e36-5c24-4642-97a3-dbe4825b6a10-tls-certs\") pod \"90966e36-5c24-4642-97a3-dbe4825b6a10\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") "
Oct 29 23:49:08.561637 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.561577    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-tmp-dir" pod "90966e36-5c24-4642-97a3-dbe4825b6a10" (UID: "90966e36-5c24-4642-97a3-dbe4825b6a10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:49:08.561637 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.561603    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-dshm\") pod \"90966e36-5c24-4642-97a3-dbe4825b6a10\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") "
Oct 29 23:49:08.561637 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.561611    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-home" pod "90966e36-5c24-4642-97a3-dbe4825b6a10" (UID: "90966e36-5c24-4642-97a3-dbe4825b6a10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:49:08.561785 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.561665    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-model-cache\") pod \"90966e36-5c24-4642-97a3-dbe4825b6a10\" (UID: \"90966e36-5c24-4642-97a3-dbe4825b6a10\") "
Oct 29 23:49:08.562061 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.562026    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-model-cache" pod "90966e36-5c24-4642-97a3-dbe4825b6a10" (UID: "90966e36-5c24-4642-97a3-dbe4825b6a10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:49:08.563695 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.562526    5428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-tmp-dir\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:49:08.563695 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.562555    5428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-home\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:49:08.563695 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.562571    5428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-model-cache\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:49:08.564796 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.564765    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90966e36-5c24-4642-97a3-dbe4825b6a10-tls-certs" pod "90966e36-5c24-4642-97a3-dbe4825b6a10" (UID: "90966e36-5c24-4642-97a3-dbe4825b6a10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:49:08.564796 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.564772    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-dshm" pod "90966e36-5c24-4642-97a3-dbe4825b6a10" (UID: "90966e36-5c24-4642-97a3-dbe4825b6a10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:49:08.565007 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.564793    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90966e36-5c24-4642-97a3-dbe4825b6a10-kube-api-access-qvpr4" pod "90966e36-5c24-4642-97a3-dbe4825b6a10" (UID: "90966e36-5c24-4642-97a3-dbe4825b6a10"). InnerVolumeSpecName "kube-api-access-qvpr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:49:08.663992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.663958    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvpr4\" (UniqueName: \"kubernetes.io/projected/90966e36-5c24-4642-97a3-dbe4825b6a10-kube-api-access-qvpr4\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:49:08.663992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.663986    5428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90966e36-5c24-4642-97a3-dbe4825b6a10-tls-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:49:08.663992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.663995    5428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90966e36-5c24-4642-97a3-dbe4825b6a10-dshm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:49:08.697586 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:08.697531    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:09.168014 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:09.167982    5428 generic.go:356] "Generic (PLEG): container finished" podID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerID="9ce48f686f5b438e9ee6617c2ec850607c5b0eecb59f9e4c3876e8770d1dd43e" exitCode=0
Oct 29 23:49:09.168185 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:09.168019    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"
Oct 29 23:49:09.168185 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:09.168059    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" event={"ID":"90966e36-5c24-4642-97a3-dbe4825b6a10","Type":"ContainerDied","Data":"9ce48f686f5b438e9ee6617c2ec850607c5b0eecb59f9e4c3876e8770d1dd43e"}
Oct 29 23:49:09.168185 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:09.168101    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct" event={"ID":"90966e36-5c24-4642-97a3-dbe4825b6a10","Type":"ContainerDied","Data":"68c89279d17b8e4fef9f38e97c5f2bc387b7ad8655d81ab764291123a948cb27"}
Oct 29 23:49:09.168185 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:09.168122    5428 scope.go:122] "RemoveContainer" containerID="9ce48f686f5b438e9ee6617c2ec850607c5b0eecb59f9e4c3876e8770d1dd43e"
Oct 29 23:49:09.178675 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:09.178655    5428 scope.go:122] "RemoveContainer" containerID="9ce48f686f5b438e9ee6617c2ec850607c5b0eecb59f9e4c3876e8770d1dd43e"
Oct 29 23:49:09.178985 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:49:09.178962    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce48f686f5b438e9ee6617c2ec850607c5b0eecb59f9e4c3876e8770d1dd43e\": container with ID starting with 9ce48f686f5b438e9ee6617c2ec850607c5b0eecb59f9e4c3876e8770d1dd43e not found: ID does not exist" containerID="9ce48f686f5b438e9ee6617c2ec850607c5b0eecb59f9e4c3876e8770d1dd43e"
Oct 29 23:49:09.179062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:09.178992    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce48f686f5b438e9ee6617c2ec850607c5b0eecb59f9e4c3876e8770d1dd43e"} err="failed to get container status \"9ce48f686f5b438e9ee6617c2ec850607c5b0eecb59f9e4c3876e8770d1dd43e\": rpc error: code = NotFound desc = could not find container \"9ce48f686f5b438e9ee6617c2ec850607c5b0eecb59f9e4c3876e8770d1dd43e\": container with ID starting with 9ce48f686f5b438e9ee6617c2ec850607c5b0eecb59f9e4c3876e8770d1dd43e not found: ID does not exist"
Oct 29 23:49:09.224434 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:09.224405    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"]
Oct 29 23:49:09.230610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:09.230585    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ed8218/premium-w4-ed8218-kserve-766c8d6bc6-t26ct"]
Oct 29 23:49:10.682270 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:10.682224    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:10.779816 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:10.779767    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" path="/var/lib/kubelet/pods/90966e36-5c24-4642-97a3-dbe4825b6a10/volumes"
Oct 29 23:49:12.117285 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:12.117239    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs" podUID="f083378a-6a46-4573-9c83-ec5af5a6e153" containerName="payload-pre-processing" containerID="cri-o://46235af407a54886a4fe4cbc279ecae379a055eb4b29f8a906e8324905a6483e" gracePeriod=25
Oct 29 23:49:12.470248 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:12.470116    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:14.283846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:14.283792    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff" podUID="5180d7a3-b01d-4639-9421-f802b9155fa2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:49:14.284243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:14.283862    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7" podUID="6f08d29e-2d26-454f-9667-8c3b99f6855d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:49:14.459505 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:14.459447    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:16.195910 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:16.195875    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:17.368059 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:49:17.368018    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:49:17.964856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:17.964813    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:18.709117 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:18.709079    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:18.846444 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:18.846415    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:19.019872 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:19.019757    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:19.399644 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:19.399610    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:19.613394 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:19.613357    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs" podUID="f083378a-6a46-4573-9c83-ec5af5a6e153" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:49:21.340399 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:21.340363    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:23.608567 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:23.608522    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:24.266291 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.266256    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7"
Oct 29 23:49:24.284639 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.284608    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f08d29e-2d26-454f-9667-8c3b99f6855d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f08d29e-2d26-454f-9667-8c3b99f6855d-plugins-config-volume\") pod \"6f08d29e-2d26-454f-9667-8c3b99f6855d\" (UID: \"6f08d29e-2d26-454f-9667-8c3b99f6855d\") "
Oct 29 23:49:24.284847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.284731    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f08d29e-2d26-454f-9667-8c3b99f6855d-kube-api-access-kx45b\" (UniqueName: \"kubernetes.io/projected/6f08d29e-2d26-454f-9667-8c3b99f6855d-kube-api-access-kx45b\") pod \"6f08d29e-2d26-454f-9667-8c3b99f6855d\" (UID: \"6f08d29e-2d26-454f-9667-8c3b99f6855d\") "
Oct 29 23:49:24.285053 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.285023    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f08d29e-2d26-454f-9667-8c3b99f6855d-plugins-config-volume" pod "6f08d29e-2d26-454f-9667-8c3b99f6855d" (UID: "6f08d29e-2d26-454f-9667-8c3b99f6855d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:49:24.285405 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.285386    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f08d29e-2d26-454f-9667-8c3b99f6855d-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:49:24.287407 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.287381    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f08d29e-2d26-454f-9667-8c3b99f6855d-kube-api-access-kx45b" pod "6f08d29e-2d26-454f-9667-8c3b99f6855d" (UID: "6f08d29e-2d26-454f-9667-8c3b99f6855d"). InnerVolumeSpecName "kube-api-access-kx45b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:49:24.296379 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.296353    5428 generic.go:356] "Generic (PLEG): container finished" podID="5180d7a3-b01d-4639-9421-f802b9155fa2" containerID="69b36197811fa0e3274a486b874a3a5fddd57f70425cacedb8ad2ca0f04b7906" exitCode=0
Oct 29 23:49:24.296504 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.296418    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff" event={"ID":"5180d7a3-b01d-4639-9421-f802b9155fa2","Type":"ContainerDied","Data":"69b36197811fa0e3274a486b874a3a5fddd57f70425cacedb8ad2ca0f04b7906"}
Oct 29 23:49:24.298288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.298262    5428 generic.go:356] "Generic (PLEG): container finished" podID="6f08d29e-2d26-454f-9667-8c3b99f6855d" containerID="c6827f0beee74dd6d3e013e2db3b060391f2f3cad34a2c6d366a1ee5fb5cf5f0" exitCode=0
Oct 29 23:49:24.298400 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.298328    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7" event={"ID":"6f08d29e-2d26-454f-9667-8c3b99f6855d","Type":"ContainerDied","Data":"c6827f0beee74dd6d3e013e2db3b060391f2f3cad34a2c6d366a1ee5fb5cf5f0"}
Oct 29 23:49:24.298400 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.298339    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7"
Oct 29 23:49:24.298400 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.298373    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7" event={"ID":"6f08d29e-2d26-454f-9667-8c3b99f6855d","Type":"ContainerDied","Data":"6c9562c2bcc5ccfe78913edda26750c5cbe56e550e30d02993c6c4cd753bccea"}
Oct 29 23:49:24.298576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.298404    5428 scope.go:122] "RemoveContainer" containerID="c6827f0beee74dd6d3e013e2db3b060391f2f3cad34a2c6d366a1ee5fb5cf5f0"
Oct 29 23:49:24.312915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.312899    5428 scope.go:122] "RemoveContainer" containerID="c6827f0beee74dd6d3e013e2db3b060391f2f3cad34a2c6d366a1ee5fb5cf5f0"
Oct 29 23:49:24.313210 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:49:24.313189    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6827f0beee74dd6d3e013e2db3b060391f2f3cad34a2c6d366a1ee5fb5cf5f0\": container with ID starting with c6827f0beee74dd6d3e013e2db3b060391f2f3cad34a2c6d366a1ee5fb5cf5f0 not found: ID does not exist" containerID="c6827f0beee74dd6d3e013e2db3b060391f2f3cad34a2c6d366a1ee5fb5cf5f0"
Oct 29 23:49:24.313288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.313217    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6827f0beee74dd6d3e013e2db3b060391f2f3cad34a2c6d366a1ee5fb5cf5f0"} err="failed to get container status \"c6827f0beee74dd6d3e013e2db3b060391f2f3cad34a2c6d366a1ee5fb5cf5f0\": rpc error: code = NotFound desc = could not find container \"c6827f0beee74dd6d3e013e2db3b060391f2f3cad34a2c6d366a1ee5fb5cf5f0\": container with ID starting with c6827f0beee74dd6d3e013e2db3b060391f2f3cad34a2c6d366a1ee5fb5cf5f0 not found: ID does not exist"
Oct 29 23:49:24.315582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.315567    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"
Oct 29 23:49:24.386556 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.386520    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5180d7a3-b01d-4639-9421-f802b9155fa2-kube-api-access-s7kp8\" (UniqueName: \"kubernetes.io/projected/5180d7a3-b01d-4639-9421-f802b9155fa2-kube-api-access-s7kp8\") pod \"5180d7a3-b01d-4639-9421-f802b9155fa2\" (UID: \"5180d7a3-b01d-4639-9421-f802b9155fa2\") "
Oct 29 23:49:24.386720 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.386582    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5180d7a3-b01d-4639-9421-f802b9155fa2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5180d7a3-b01d-4639-9421-f802b9155fa2-plugins-config-volume\") pod \"5180d7a3-b01d-4639-9421-f802b9155fa2\" (UID: \"5180d7a3-b01d-4639-9421-f802b9155fa2\") "
Oct 29 23:49:24.387190 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.387162    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5180d7a3-b01d-4639-9421-f802b9155fa2-plugins-config-volume" pod "5180d7a3-b01d-4639-9421-f802b9155fa2" (UID: "5180d7a3-b01d-4639-9421-f802b9155fa2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:49:24.387419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.387391    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5180d7a3-b01d-4639-9421-f802b9155fa2-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:49:24.387419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.387419    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx45b\" (UniqueName: \"kubernetes.io/projected/6f08d29e-2d26-454f-9667-8c3b99f6855d-kube-api-access-kx45b\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:49:24.388984 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.388957    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5180d7a3-b01d-4639-9421-f802b9155fa2-kube-api-access-s7kp8" pod "5180d7a3-b01d-4639-9421-f802b9155fa2" (UID: "5180d7a3-b01d-4639-9421-f802b9155fa2"). InnerVolumeSpecName "kube-api-access-s7kp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:49:24.418941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.418614    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7"]
Oct 29 23:49:24.425254 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.425231    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ed8218-7c84cc766b-gw4k7"]
Oct 29 23:49:24.488599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.488523    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7kp8\" (UniqueName: \"kubernetes.io/projected/5180d7a3-b01d-4639-9421-f802b9155fa2-kube-api-access-s7kp8\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:49:24.778493 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.778412    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f08d29e-2d26-454f-9667-8c3b99f6855d" path="/var/lib/kubelet/pods/6f08d29e-2d26-454f-9667-8c3b99f6855d/volumes"
Oct 29 23:49:24.906764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:24.906729    5428 scope.go:122] "RemoveContainer" containerID="69b36197811fa0e3274a486b874a3a5fddd57f70425cacedb8ad2ca0f04b7906"
Oct 29 23:49:25.283230 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:25.283189    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff" podUID="5180d7a3-b01d-4639-9421-f802b9155fa2" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.73:50052\" within 1s: context deadline exceeded"
Oct 29 23:49:25.283411 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:49:25.283248    5428 logging.go:55] [core] [Channel #25379 SubChannel #25380]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.73:50052", ServerName: "10.133.0.73:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.73:50052: operation was canceled"
Oct 29 23:49:25.283411 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:25.283272    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"
Oct 29 23:49:25.305050 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:25.305020    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"
Oct 29 23:49:25.305210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:25.305029    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff" event={"ID":"5180d7a3-b01d-4639-9421-f802b9155fa2","Type":"ContainerDied","Data":"ab864367a05556056a1e1b87aede485a4a3222edf823b27a5c19c3abb045e186"}
Oct 29 23:49:25.397374 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:25.397343    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"]
Oct 29 23:49:25.420488 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:25.420458    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ed8218-784746bc67-f4cff"]
Oct 29 23:49:25.659608 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:25.659572    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:26.778543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:26.778508    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5180d7a3-b01d-4639-9421-f802b9155fa2" path="/var/lib/kubelet/pods/5180d7a3-b01d-4639-9421-f802b9155fa2/volumes"
Oct 29 23:49:27.337147 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:27.337103    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:27.481279 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:27.481245    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:27.795168 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:27.795098    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:28.036519 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:28.036482    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:29.613608 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:29.613559    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs" podUID="f083378a-6a46-4573-9c83-ec5af5a6e153" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:49:29.943374 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:29.943335    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:32.425587 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:32.425510    5428 generic.go:356] "Generic (PLEG): container finished" podID="f083378a-6a46-4573-9c83-ec5af5a6e153" containerID="46235af407a54886a4fe4cbc279ecae379a055eb4b29f8a906e8324905a6483e" exitCode=0
Oct 29 23:49:32.426186 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:32.425914    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs" event={"ID":"f083378a-6a46-4573-9c83-ec5af5a6e153","Type":"ContainerDied","Data":"46235af407a54886a4fe4cbc279ecae379a055eb4b29f8a906e8324905a6483e"}
Oct 29 23:49:32.430006 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:32.429985    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs"
Oct 29 23:49:32.475670 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:32.475641    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:32.528565 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:32.526157    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f083378a-6a46-4573-9c83-ec5af5a6e153-kube-api-access-k4lvn\" (UniqueName: \"kubernetes.io/projected/f083378a-6a46-4573-9c83-ec5af5a6e153-kube-api-access-k4lvn\") pod \"f083378a-6a46-4573-9c83-ec5af5a6e153\" (UID: \"f083378a-6a46-4573-9c83-ec5af5a6e153\") "
Oct 29 23:49:32.528565 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:32.526352    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f083378a-6a46-4573-9c83-ec5af5a6e153-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f083378a-6a46-4573-9c83-ec5af5a6e153-plugins-config-volume\") pod \"f083378a-6a46-4573-9c83-ec5af5a6e153\" (UID: \"f083378a-6a46-4573-9c83-ec5af5a6e153\") "
Oct 29 23:49:32.531544 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:32.531503    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f083378a-6a46-4573-9c83-ec5af5a6e153-plugins-config-volume" pod "f083378a-6a46-4573-9c83-ec5af5a6e153" (UID: "f083378a-6a46-4573-9c83-ec5af5a6e153"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:49:32.534653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:32.534515    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f083378a-6a46-4573-9c83-ec5af5a6e153-kube-api-access-k4lvn" pod "f083378a-6a46-4573-9c83-ec5af5a6e153" (UID: "f083378a-6a46-4573-9c83-ec5af5a6e153"). InnerVolumeSpecName "kube-api-access-k4lvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:49:32.631222 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:32.631188    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4lvn\" (UniqueName: \"kubernetes.io/projected/f083378a-6a46-4573-9c83-ec5af5a6e153-kube-api-access-k4lvn\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:49:32.631222 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:32.631216    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f083378a-6a46-4573-9c83-ec5af5a6e153-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:49:32.665839 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:32.665755    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:33.437473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:33.437444    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs"
Oct 29 23:49:33.437939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:33.437489    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs" event={"ID":"f083378a-6a46-4573-9c83-ec5af5a6e153","Type":"ContainerDied","Data":"3e14c19e745486edf682e885112bbd58e588d4bbaa34b15fef22c86cf189d4dd"}
Oct 29 23:49:33.437939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:33.437557    5428 scope.go:122] "RemoveContainer" containerID="46235af407a54886a4fe4cbc279ecae379a055eb4b29f8a906e8324905a6483e"
Oct 29 23:49:33.503912 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:33.503879    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs"]
Oct 29 23:49:33.514461 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:33.514434    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e57216-cd5787db7-dl7gs"]
Oct 29 23:49:33.692883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:33.692792    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:33.940642 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:33.940608    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:34.242984 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:34.242943    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:34.449313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:34.449281    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:34.760501 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:34.760456    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:34.804203 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:34.804168    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f083378a-6a46-4573-9c83-ec5af5a6e153" path="/var/lib/kubelet/pods/f083378a-6a46-4573-9c83-ec5af5a6e153/volumes"
Oct 29 23:49:35.169733 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:35.169695    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:35.516964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:35.516880    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:35.902710 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:35.902672    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:37.544176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:37.544139    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:38.882038 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:38.882007    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6"]
Oct 29 23:49:39.072848 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:39.072791    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:39.245418 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:39.245325    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:39.521970 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:39.521878    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:41.611228 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:41.611194    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:42.094282 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:42.094250    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:42.303963 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:42.303934    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:43.623764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:43.623709    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:43.842933 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:43.842894    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:43.882325 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:43.882243    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6" podUID="85670e7f-43f1-4f46-b691-88beee106a99" containerName="payload-processing" containerID="cri-o://4f44d3e932ee51db2557695c5371a2000e660d58276d1429b7a331b148bd17e5" gracePeriod=25
Oct 29 23:49:44.188816 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:44.188777    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6"
Oct 29 23:49:44.319822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:44.316665    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85670e7f-43f1-4f46-b691-88beee106a99-kube-api-access-z5nxd\" (UniqueName: \"kubernetes.io/projected/85670e7f-43f1-4f46-b691-88beee106a99-kube-api-access-z5nxd\") pod \"85670e7f-43f1-4f46-b691-88beee106a99\" (UID: \"85670e7f-43f1-4f46-b691-88beee106a99\") "
Oct 29 23:49:44.319822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:44.316855    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85670e7f-43f1-4f46-b691-88beee106a99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85670e7f-43f1-4f46-b691-88beee106a99-plugins-config-volume\") pod \"85670e7f-43f1-4f46-b691-88beee106a99\" (UID: \"85670e7f-43f1-4f46-b691-88beee106a99\") "
Oct 29 23:49:44.319822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:44.317963    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85670e7f-43f1-4f46-b691-88beee106a99-plugins-config-volume" pod "85670e7f-43f1-4f46-b691-88beee106a99" (UID: "85670e7f-43f1-4f46-b691-88beee106a99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:49:44.320966 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:44.320927    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85670e7f-43f1-4f46-b691-88beee106a99-kube-api-access-z5nxd" pod "85670e7f-43f1-4f46-b691-88beee106a99" (UID: "85670e7f-43f1-4f46-b691-88beee106a99"). InnerVolumeSpecName "kube-api-access-z5nxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:49:44.418481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:44.418397    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5nxd\" (UniqueName: \"kubernetes.io/projected/85670e7f-43f1-4f46-b691-88beee106a99-kube-api-access-z5nxd\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:49:44.418481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:44.418428    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85670e7f-43f1-4f46-b691-88beee106a99-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:49:44.553866 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:44.553788    5428 generic.go:356] "Generic (PLEG): container finished" podID="85670e7f-43f1-4f46-b691-88beee106a99" containerID="4f44d3e932ee51db2557695c5371a2000e660d58276d1429b7a331b148bd17e5" exitCode=0
Oct 29 23:49:44.553866 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:44.553852    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6"
Oct 29 23:49:44.553866 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:44.553855    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6" event={"ID":"85670e7f-43f1-4f46-b691-88beee106a99","Type":"ContainerDied","Data":"4f44d3e932ee51db2557695c5371a2000e660d58276d1429b7a331b148bd17e5"}
Oct 29 23:49:44.554120 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:44.553923    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6" event={"ID":"85670e7f-43f1-4f46-b691-88beee106a99","Type":"ContainerDied","Data":"a21e1b799c4852f8bc49c896ad573905a51c3c3abdd87d8b7175281a38cbfe19"}
Oct 29 23:49:44.554120 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:44.553954    5428 scope.go:122] "RemoveContainer" containerID="4f44d3e932ee51db2557695c5371a2000e660d58276d1429b7a331b148bd17e5"
Oct 29 23:49:44.565451 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:44.565428    5428 scope.go:122] "RemoveContainer" containerID="4f44d3e932ee51db2557695c5371a2000e660d58276d1429b7a331b148bd17e5"
Oct 29 23:49:44.565736 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:49:44.565714    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f44d3e932ee51db2557695c5371a2000e660d58276d1429b7a331b148bd17e5\": container with ID starting with 4f44d3e932ee51db2557695c5371a2000e660d58276d1429b7a331b148bd17e5 not found: ID does not exist" containerID="4f44d3e932ee51db2557695c5371a2000e660d58276d1429b7a331b148bd17e5"
Oct 29 23:49:44.565847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:44.565742    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f44d3e932ee51db2557695c5371a2000e660d58276d1429b7a331b148bd17e5"} err="failed to get container status \"4f44d3e932ee51db2557695c5371a2000e660d58276d1429b7a331b148bd17e5\": rpc error: code = NotFound desc = could not find container \"4f44d3e932ee51db2557695c5371a2000e660d58276d1429b7a331b148bd17e5\": container with ID starting with 4f44d3e932ee51db2557695c5371a2000e660d58276d1429b7a331b148bd17e5 not found: ID does not exist"
Oct 29 23:49:44.618783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:44.618750    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6"]
Oct 29 23:49:44.625920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:44.625890    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-2591f6-5f6bbc8c78-tt2d6"]
Oct 29 23:49:44.782210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:44.782132    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85670e7f-43f1-4f46-b691-88beee106a99" path="/var/lib/kubelet/pods/85670e7f-43f1-4f46-b691-88beee106a99/volumes"
Oct 29 23:49:46.305166 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:46.305128    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:47.368009 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:49:47.367965    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:49:48.272526 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:48.272486    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:50.102839 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:50.102784    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:52.162361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:52.162327    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:54.132021 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:54.131980    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:56.118924 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:56.118890    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:58.020993 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:58.020946    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:58.162502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:58.162464    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"]
Oct 29 23:49:58.217381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:58.217346    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:49:59.947702 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:49:59.947666    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:01.968043 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:01.968010    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:04.066485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:04.063022    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:05.943412 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:05.943376    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:06.790635 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:06.790597    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:06.967844 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:06.967778    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:07.136388 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:07.136356    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:07.463070 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:07.462970    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:09.317085 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:09.317050    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:10.763479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:10.763444    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:10.899670 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:10.899634    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:11.197431 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:11.197400    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:12.928952 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:12.928920    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:13.186034 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.185944    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t" podUID="ad7987fc-c841-4693-94e3-36c855add12c" containerName="main" containerID="cri-o://2e707cbf2defea8b3199d2015195f8ddf339927c2b868a73f17624e4e061a1ff" gracePeriod=45
Oct 29 23:50:13.460310 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.460287    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:50:13.547362 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.547325    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad7987fc-c841-4693-94e3-36c855add12c-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad7987fc-c841-4693-94e3-36c855add12c-tls-certs\") pod \"ad7987fc-c841-4693-94e3-36c855add12c\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") "
Oct 29 23:50:13.547533 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.547378    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad7987fc-c841-4693-94e3-36c855add12c-kube-api-access-9v472\" (UniqueName: \"kubernetes.io/projected/ad7987fc-c841-4693-94e3-36c855add12c-kube-api-access-9v472\") pod \"ad7987fc-c841-4693-94e3-36c855add12c\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") "
Oct 29 23:50:13.547533 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.547412    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-home\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-home\") pod \"ad7987fc-c841-4693-94e3-36c855add12c\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") "
Oct 29 23:50:13.547665 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.547608    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-model-cache\") pod \"ad7987fc-c841-4693-94e3-36c855add12c\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") "
Oct 29 23:50:13.547727 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.547664    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-dshm\") pod \"ad7987fc-c841-4693-94e3-36c855add12c\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") "
Oct 29 23:50:13.547727 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.547701    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-tmp-dir\") pod \"ad7987fc-c841-4693-94e3-36c855add12c\" (UID: \"ad7987fc-c841-4693-94e3-36c855add12c\") "
Oct 29 23:50:13.547878 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.547791    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-home" pod "ad7987fc-c841-4693-94e3-36c855add12c" (UID: "ad7987fc-c841-4693-94e3-36c855add12c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:50:13.547988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.547957    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-model-cache" pod "ad7987fc-c841-4693-94e3-36c855add12c" (UID: "ad7987fc-c841-4693-94e3-36c855add12c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:50:13.548279 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.548222    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-tmp-dir" pod "ad7987fc-c841-4693-94e3-36c855add12c" (UID: "ad7987fc-c841-4693-94e3-36c855add12c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:50:13.548427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.548415    5428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-home\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:13.548500 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.548436    5428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-model-cache\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:13.548500 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.548450    5428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-tmp-dir\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:13.549971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.549949    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-dshm" pod "ad7987fc-c841-4693-94e3-36c855add12c" (UID: "ad7987fc-c841-4693-94e3-36c855add12c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:50:13.550224 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.550200    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7987fc-c841-4693-94e3-36c855add12c-kube-api-access-9v472" pod "ad7987fc-c841-4693-94e3-36c855add12c" (UID: "ad7987fc-c841-4693-94e3-36c855add12c"). InnerVolumeSpecName "kube-api-access-9v472". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:50:13.550430 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.550407    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad7987fc-c841-4693-94e3-36c855add12c-tls-certs" pod "ad7987fc-c841-4693-94e3-36c855add12c" (UID: "ad7987fc-c841-4693-94e3-36c855add12c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:50:13.650290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.650254    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v472\" (UniqueName: \"kubernetes.io/projected/ad7987fc-c841-4693-94e3-36c855add12c-kube-api-access-9v472\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:13.650290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.650285    5428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad7987fc-c841-4693-94e3-36c855add12c-dshm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:13.650290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.650297    5428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad7987fc-c841-4693-94e3-36c855add12c-tls-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:13.788159 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.788081    5428 generic.go:356] "Generic (PLEG): container finished" podID="ad7987fc-c841-4693-94e3-36c855add12c" containerID="2e707cbf2defea8b3199d2015195f8ddf339927c2b868a73f17624e4e061a1ff" exitCode=0
Oct 29 23:50:13.788159 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.788118    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t" event={"ID":"ad7987fc-c841-4693-94e3-36c855add12c","Type":"ContainerDied","Data":"2e707cbf2defea8b3199d2015195f8ddf339927c2b868a73f17624e4e061a1ff"}
Oct 29 23:50:13.788159 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.788156    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t" event={"ID":"ad7987fc-c841-4693-94e3-36c855add12c","Type":"ContainerDied","Data":"54a47aa5343230452c9ec6484924ef59b803cbff4ab7bc5fab9930d22eec8216"}
Oct 29 23:50:13.788496 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.788174    5428 scope.go:122] "RemoveContainer" containerID="2e707cbf2defea8b3199d2015195f8ddf339927c2b868a73f17624e4e061a1ff"
Oct 29 23:50:13.788496 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.788126    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"
Oct 29 23:50:13.799758 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.799740    5428 scope.go:122] "RemoveContainer" containerID="2e707cbf2defea8b3199d2015195f8ddf339927c2b868a73f17624e4e061a1ff"
Oct 29 23:50:13.800068 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:50:13.800039    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e707cbf2defea8b3199d2015195f8ddf339927c2b868a73f17624e4e061a1ff\": container with ID starting with 2e707cbf2defea8b3199d2015195f8ddf339927c2b868a73f17624e4e061a1ff not found: ID does not exist" containerID="2e707cbf2defea8b3199d2015195f8ddf339927c2b868a73f17624e4e061a1ff"
Oct 29 23:50:13.800175 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.800072    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e707cbf2defea8b3199d2015195f8ddf339927c2b868a73f17624e4e061a1ff"} err="failed to get container status \"2e707cbf2defea8b3199d2015195f8ddf339927c2b868a73f17624e4e061a1ff\": rpc error: code = NotFound desc = could not find container \"2e707cbf2defea8b3199d2015195f8ddf339927c2b868a73f17624e4e061a1ff\": container with ID starting with 2e707cbf2defea8b3199d2015195f8ddf339927c2b868a73f17624e4e061a1ff not found: ID does not exist"
Oct 29 23:50:13.844066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.844038    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"]
Oct 29 23:50:13.856307 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:13.856281    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-750972/test-model-750972-kserve-56f665676c-6s56t"]
Oct 29 23:50:14.026956 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:14.026904    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:14.147697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:14.147666    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:14.392896 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:14.392865    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:14.778250 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:14.778206    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad7987fc-c841-4693-94e3-36c855add12c" path="/var/lib/kubelet/pods/ad7987fc-c841-4693-94e3-36c855add12c/volumes"
Oct 29 23:50:14.998430 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:14.998396    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:16.724329 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:16.724299    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:17.368285 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:50:17.368246    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:50:18.592996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:18.592945    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:19.452683 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:19.452649    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:20.639793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:20.639757    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:22.345003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.344966    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"]
Oct 29 23:50:22.345618 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.345179    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s" podUID="afe8000b-075b-4052-b38c-fe2a222b8306" containerName="maas-api" containerID="cri-o://ea224f3cecd5286738db59b1a2496130dcc2fd05b7762a7166615ca160dc373e" gracePeriod=30
Oct 29 23:50:22.436941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.436901    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:22.671174 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.671142    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"
Oct 29 23:50:22.710147 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.710115    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe8000b-075b-4052-b38c-fe2a222b8306-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afe8000b-075b-4052-b38c-fe2a222b8306-openshift-service-ca\") pod \"afe8000b-075b-4052-b38c-fe2a222b8306\" (UID: \"afe8000b-075b-4052-b38c-fe2a222b8306\") "
Oct 29 23:50:22.710322 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.710287    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe8000b-075b-4052-b38c-fe2a222b8306-kube-api-access-spw7l\" (UniqueName: \"kubernetes.io/projected/afe8000b-075b-4052-b38c-fe2a222b8306-kube-api-access-spw7l\") pod \"afe8000b-075b-4052-b38c-fe2a222b8306\" (UID: \"afe8000b-075b-4052-b38c-fe2a222b8306\") "
Oct 29 23:50:22.710406 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.710376    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afe8000b-075b-4052-b38c-fe2a222b8306-metrics-certs\" (UniqueName: \"kubernetes.io/secret/afe8000b-075b-4052-b38c-fe2a222b8306-metrics-certs\") pod \"afe8000b-075b-4052-b38c-fe2a222b8306\" (UID: \"afe8000b-075b-4052-b38c-fe2a222b8306\") "
Oct 29 23:50:22.710472 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.710424    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afe8000b-075b-4052-b38c-fe2a222b8306-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afe8000b-075b-4052-b38c-fe2a222b8306-maas-api-tls\") pod \"afe8000b-075b-4052-b38c-fe2a222b8306\" (UID: \"afe8000b-075b-4052-b38c-fe2a222b8306\") "
Oct 29 23:50:22.710535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.710472    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe8000b-075b-4052-b38c-fe2a222b8306-openshift-service-ca" pod "afe8000b-075b-4052-b38c-fe2a222b8306" (UID: "afe8000b-075b-4052-b38c-fe2a222b8306"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:50:22.711005 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.710976    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afe8000b-075b-4052-b38c-fe2a222b8306-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:22.712496 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.712470    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe8000b-075b-4052-b38c-fe2a222b8306-kube-api-access-spw7l" pod "afe8000b-075b-4052-b38c-fe2a222b8306" (UID: "afe8000b-075b-4052-b38c-fe2a222b8306"). InnerVolumeSpecName "kube-api-access-spw7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:50:22.712595 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.712470    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe8000b-075b-4052-b38c-fe2a222b8306-metrics-certs" pod "afe8000b-075b-4052-b38c-fe2a222b8306" (UID: "afe8000b-075b-4052-b38c-fe2a222b8306"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:50:22.712595 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.712538    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe8000b-075b-4052-b38c-fe2a222b8306-maas-api-tls" pod "afe8000b-075b-4052-b38c-fe2a222b8306" (UID: "afe8000b-075b-4052-b38c-fe2a222b8306"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:50:22.812759 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.812731    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spw7l\" (UniqueName: \"kubernetes.io/projected/afe8000b-075b-4052-b38c-fe2a222b8306-kube-api-access-spw7l\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:22.812759 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.812758    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afe8000b-075b-4052-b38c-fe2a222b8306-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:22.812759 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.812772    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afe8000b-075b-4052-b38c-fe2a222b8306-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:22.859133 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.859099    5428 generic.go:356] "Generic (PLEG): container finished" podID="afe8000b-075b-4052-b38c-fe2a222b8306" containerID="ea224f3cecd5286738db59b1a2496130dcc2fd05b7762a7166615ca160dc373e" exitCode=0
Oct 29 23:50:22.859318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.859211    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s" event={"ID":"afe8000b-075b-4052-b38c-fe2a222b8306","Type":"ContainerDied","Data":"ea224f3cecd5286738db59b1a2496130dcc2fd05b7762a7166615ca160dc373e"}
Oct 29 23:50:22.859318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.859249    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s" event={"ID":"afe8000b-075b-4052-b38c-fe2a222b8306","Type":"ContainerDied","Data":"4e1a6c612fe57368a34e007d2d44bc39fcb5e0e752343254e9007f5066605c8c"}
Oct 29 23:50:22.859318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.859269    5428 scope.go:122] "RemoveContainer" containerID="ea224f3cecd5286738db59b1a2496130dcc2fd05b7762a7166615ca160dc373e"
Oct 29 23:50:22.859486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.859368    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"
Oct 29 23:50:22.879871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.879793    5428 scope.go:122] "RemoveContainer" containerID="ea224f3cecd5286738db59b1a2496130dcc2fd05b7762a7166615ca160dc373e"
Oct 29 23:50:22.880208 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:50:22.880182    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea224f3cecd5286738db59b1a2496130dcc2fd05b7762a7166615ca160dc373e\": container with ID starting with ea224f3cecd5286738db59b1a2496130dcc2fd05b7762a7166615ca160dc373e not found: ID does not exist" containerID="ea224f3cecd5286738db59b1a2496130dcc2fd05b7762a7166615ca160dc373e"
Oct 29 23:50:22.880353 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.880219    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea224f3cecd5286738db59b1a2496130dcc2fd05b7762a7166615ca160dc373e"} err="failed to get container status \"ea224f3cecd5286738db59b1a2496130dcc2fd05b7762a7166615ca160dc373e\": rpc error: code = NotFound desc = could not find container \"ea224f3cecd5286738db59b1a2496130dcc2fd05b7762a7166615ca160dc373e\": container with ID starting with ea224f3cecd5286738db59b1a2496130dcc2fd05b7762a7166615ca160dc373e not found: ID does not exist"
Oct 29 23:50:22.933582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.933541    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"]
Oct 29 23:50:22.942199 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:22.942163    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s"]
Oct 29 23:50:23.224458 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:23.224363    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:23.353787 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:23.353751    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:23.533916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:23.533771    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:23.887373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:23.887335    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:24.777915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:24.777882    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe8000b-075b-4052-b38c-fe2a222b8306" path="/var/lib/kubelet/pods/afe8000b-075b-4052-b38c-fe2a222b8306/volumes"
Oct 29 23:50:25.281589 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:25.281548    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:25.607526 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:25.607479    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-2591f6-659f7757c9-xtg5s" podUID="afe8000b-075b-4052-b38c-fe2a222b8306" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.211:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:50:26.802979 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:26.802949    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"]
Oct 29 23:50:26.803428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:26.803197    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8" podUID="c1dae43c-236e-4510-a6b9-0ffa73efe69f" containerName="maas-api" containerID="cri-o://ed2771edf8dd43d39cac77fd9fa7370bf6c1ff3c1a57547c8a057bf1792e60a8" gracePeriod=30
Oct 29 23:50:26.882776 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:26.882740    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:27.130403 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.130373    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"
Oct 29 23:50:27.277251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.277215    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1dae43c-236e-4510-a6b9-0ffa73efe69f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dae43c-236e-4510-a6b9-0ffa73efe69f-maas-api-tls\") pod \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\" (UID: \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\") "
Oct 29 23:50:27.277457 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.277349    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1dae43c-236e-4510-a6b9-0ffa73efe69f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dae43c-236e-4510-a6b9-0ffa73efe69f-openshift-service-ca\") pod \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\" (UID: \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\") "
Oct 29 23:50:27.277457 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.277396    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1dae43c-236e-4510-a6b9-0ffa73efe69f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dae43c-236e-4510-a6b9-0ffa73efe69f-metrics-certs\") pod \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\" (UID: \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\") "
Oct 29 23:50:27.277603 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.277509    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1dae43c-236e-4510-a6b9-0ffa73efe69f-kube-api-access-49vl5\" (UniqueName: \"kubernetes.io/projected/c1dae43c-236e-4510-a6b9-0ffa73efe69f-kube-api-access-49vl5\") pod \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\" (UID: \"c1dae43c-236e-4510-a6b9-0ffa73efe69f\") "
Oct 29 23:50:27.278297 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.278262    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1dae43c-236e-4510-a6b9-0ffa73efe69f-openshift-service-ca" pod "c1dae43c-236e-4510-a6b9-0ffa73efe69f" (UID: "c1dae43c-236e-4510-a6b9-0ffa73efe69f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:50:27.279619 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.279586    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1dae43c-236e-4510-a6b9-0ffa73efe69f-maas-api-tls" pod "c1dae43c-236e-4510-a6b9-0ffa73efe69f" (UID: "c1dae43c-236e-4510-a6b9-0ffa73efe69f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:50:27.279735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.279633    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1dae43c-236e-4510-a6b9-0ffa73efe69f-metrics-certs" pod "c1dae43c-236e-4510-a6b9-0ffa73efe69f" (UID: "c1dae43c-236e-4510-a6b9-0ffa73efe69f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:50:27.279735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.279642    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dae43c-236e-4510-a6b9-0ffa73efe69f-kube-api-access-49vl5" pod "c1dae43c-236e-4510-a6b9-0ffa73efe69f" (UID: "c1dae43c-236e-4510-a6b9-0ffa73efe69f"). InnerVolumeSpecName "kube-api-access-49vl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:50:27.381974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.381928    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dae43c-236e-4510-a6b9-0ffa73efe69f-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:27.382300 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.382276    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dae43c-236e-4510-a6b9-0ffa73efe69f-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:27.382439 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.382423    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dae43c-236e-4510-a6b9-0ffa73efe69f-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:27.382538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.382526    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49vl5\" (UniqueName: \"kubernetes.io/projected/c1dae43c-236e-4510-a6b9-0ffa73efe69f-kube-api-access-49vl5\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:27.472648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.472598    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:27.638020 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.637939    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:27.786262 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.786222    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:27.897571 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.897491    5428 generic.go:356] "Generic (PLEG): container finished" podID="c1dae43c-236e-4510-a6b9-0ffa73efe69f" containerID="ed2771edf8dd43d39cac77fd9fa7370bf6c1ff3c1a57547c8a057bf1792e60a8" exitCode=0
Oct 29 23:50:27.897571 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.897533    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"
Oct 29 23:50:27.898039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.897574    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8" event={"ID":"c1dae43c-236e-4510-a6b9-0ffa73efe69f","Type":"ContainerDied","Data":"ed2771edf8dd43d39cac77fd9fa7370bf6c1ff3c1a57547c8a057bf1792e60a8"}
Oct 29 23:50:27.898039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.897626    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8" event={"ID":"c1dae43c-236e-4510-a6b9-0ffa73efe69f","Type":"ContainerDied","Data":"bd4c1a4c1452fcf08a0dbc94f3c20a384441cc149c3d2a6f693aa824d656578b"}
Oct 29 23:50:27.898039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.897649    5428 scope.go:122] "RemoveContainer" containerID="ed2771edf8dd43d39cac77fd9fa7370bf6c1ff3c1a57547c8a057bf1792e60a8"
Oct 29 23:50:27.910168 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.910152    5428 scope.go:122] "RemoveContainer" containerID="ed2771edf8dd43d39cac77fd9fa7370bf6c1ff3c1a57547c8a057bf1792e60a8"
Oct 29 23:50:27.910440 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:50:27.910419    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed2771edf8dd43d39cac77fd9fa7370bf6c1ff3c1a57547c8a057bf1792e60a8\": container with ID starting with ed2771edf8dd43d39cac77fd9fa7370bf6c1ff3c1a57547c8a057bf1792e60a8 not found: ID does not exist" containerID="ed2771edf8dd43d39cac77fd9fa7370bf6c1ff3c1a57547c8a057bf1792e60a8"
Oct 29 23:50:27.910517 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.910447    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2771edf8dd43d39cac77fd9fa7370bf6c1ff3c1a57547c8a057bf1792e60a8"} err="failed to get container status \"ed2771edf8dd43d39cac77fd9fa7370bf6c1ff3c1a57547c8a057bf1792e60a8\": rpc error: code = NotFound desc = could not find container \"ed2771edf8dd43d39cac77fd9fa7370bf6c1ff3c1a57547c8a057bf1792e60a8\": container with ID starting with ed2771edf8dd43d39cac77fd9fa7370bf6c1ff3c1a57547c8a057bf1792e60a8 not found: ID does not exist"
Oct 29 23:50:27.958384 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.958345    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"]
Oct 29 23:50:27.969913 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:27.969874    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e57216-6667f455f-2fbn8"]
Oct 29 23:50:28.233577 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:28.233488    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:28.779313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:28.779284    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1dae43c-236e-4510-a6b9-0ffa73efe69f" path="/var/lib/kubelet/pods/c1dae43c-236e-4510-a6b9-0ffa73efe69f/volumes"
Oct 29 23:50:29.996386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:29.996348    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:30.229507 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:30.229471    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn"]
Oct 29 23:50:30.396215 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:30.396184    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:30.523595 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:30.523558    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:31.787281 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:31.787250    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:31.927659 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:31.927619    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:32.166732 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:32.166691    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:33.941711 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:33.941670    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:34.172539 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:34.172510    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:34.653490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:34.653442    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:35.100984 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:35.100897    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:35.229856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:35.229773    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn" podUID="22e5475e-1e3a-487e-a50c-97851f4e845d" containerName="payload-processing" containerID="cri-o://e396f1ea5cb5d84a2cf589e780a3a939cec7e7de97341e03df45a12b91c78bc0" gracePeriod=25
Oct 29 23:50:35.244511 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:35.244482    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:35.421504 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:35.421471    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"]
Oct 29 23:50:35.422012 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:35.421978    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm" podUID="9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" containerName="istio-proxy" containerID="cri-o://d35dfdc14fc08f53212ad77ef76163513c0108fd2e4ff08315b0b13d2bc643d3" gracePeriod=30
Oct 29 23:50:35.492663 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:35.492626    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:36.090291 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:36.090258    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:37.611186 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:37.611145    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:38.343104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:38.343068    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:38.482259 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:38.482221    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:38.608061 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:38.607981    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:39.013633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:39.013589    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:40.399246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.399216    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:40.713322 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.713295    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 23:50:40.770984 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.770870    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-kube-api-access-wjrxv\" (UniqueName: \"kubernetes.io/projected/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-kube-api-access-wjrxv\") pod \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") "
Oct 29 23:50:40.770984 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.770920    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-credential-socket\") pod \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") "
Oct 29 23:50:40.771176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.771151    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-envoy\") pod \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") "
Oct 29 23:50:40.771247 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.771191    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-podinfo\") pod \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") "
Oct 29 23:50:40.771247 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.771218    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-data\") pod \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") "
Oct 29 23:50:40.771368 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.771284    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-workload-socket\") pod \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") "
Oct 29 23:50:40.771368 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.771334    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-workload-certs\") pod \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") "
Oct 29 23:50:40.771490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.771374    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istiod-ca-cert\") pod \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") "
Oct 29 23:50:40.771490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.771414    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-token\" (UniqueName: \"kubernetes.io/projected/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-token\") pod \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\" (UID: \"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9\") "
Oct 29 23:50:40.771490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.771437    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-credential-socket" pod "9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" (UID: "9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:50:40.771677 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.771655    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-workload-socket" pod "9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" (UID: "9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:50:40.771984 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.771964    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-data" pod "9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" (UID: "9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:50:40.772112 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.771993    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:40.772258 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.772241    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:40.772350 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.772189    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-workload-certs" pod "9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" (UID: "9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:50:40.774143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.774122    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istiod-ca-cert" pod "9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" (UID: "9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:50:40.774739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.774709    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-kube-api-access-wjrxv" pod "9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" (UID: "9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9"). InnerVolumeSpecName "kube-api-access-wjrxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:50:40.774877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.774791    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-envoy" pod "9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" (UID: "9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:50:40.775351 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.775325    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-podinfo" pod "9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" (UID: "9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:50:40.777533 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.777510    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-token" pod "9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" (UID: "9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:50:40.873989 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.873959    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:40.873989 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.873988    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:40.873989 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.873997    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:40.874217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.874006    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:40.874217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.874015    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:40.874217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.874022    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:40.874217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:40.874030    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjrxv\" (UniqueName: \"kubernetes.io/projected/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9-kube-api-access-wjrxv\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:41.004337 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:41.004242    5428 generic.go:356] "Generic (PLEG): container finished" podID="9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" containerID="d35dfdc14fc08f53212ad77ef76163513c0108fd2e4ff08315b0b13d2bc643d3" exitCode=0
Oct 29 23:50:41.004337 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:41.004299    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"
Oct 29 23:50:41.004337 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:41.004325    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm" event={"ID":"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9","Type":"ContainerDied","Data":"d35dfdc14fc08f53212ad77ef76163513c0108fd2e4ff08315b0b13d2bc643d3"}
Oct 29 23:50:41.004632 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:41.004381    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm" event={"ID":"9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9","Type":"ContainerDied","Data":"be6d140dee660ece155bbbfc14587a93f788aeb1ede8a2c5388b046d65e1a833"}
Oct 29 23:50:41.004632 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:41.004406    5428 scope.go:122] "RemoveContainer" containerID="d35dfdc14fc08f53212ad77ef76163513c0108fd2e4ff08315b0b13d2bc643d3"
Oct 29 23:50:41.016740 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:41.016717    5428 scope.go:122] "RemoveContainer" containerID="d35dfdc14fc08f53212ad77ef76163513c0108fd2e4ff08315b0b13d2bc643d3"
Oct 29 23:50:41.017056 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:50:41.017025    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35dfdc14fc08f53212ad77ef76163513c0108fd2e4ff08315b0b13d2bc643d3\": container with ID starting with d35dfdc14fc08f53212ad77ef76163513c0108fd2e4ff08315b0b13d2bc643d3 not found: ID does not exist" containerID="d35dfdc14fc08f53212ad77ef76163513c0108fd2e4ff08315b0b13d2bc643d3"
Oct 29 23:50:41.017253 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:41.017226    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35dfdc14fc08f53212ad77ef76163513c0108fd2e4ff08315b0b13d2bc643d3"} err="failed to get container status \"d35dfdc14fc08f53212ad77ef76163513c0108fd2e4ff08315b0b13d2bc643d3\": rpc error: code = NotFound desc = could not find container \"d35dfdc14fc08f53212ad77ef76163513c0108fd2e4ff08315b0b13d2bc643d3\": container with ID starting with d35dfdc14fc08f53212ad77ef76163513c0108fd2e4ff08315b0b13d2bc643d3 not found: ID does not exist"
Oct 29 23:50:41.060157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:41.060124    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"]
Oct 29 23:50:41.073091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:41.073061    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e57216-openshift-default-59dd4b5cd9-mmpgm"]
Oct 29 23:50:42.147890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:42.147858    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:42.198173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:42.198130    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn" podUID="22e5475e-1e3a-487e-a50c-97851f4e845d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:50:42.779011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:42.778975    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" path="/var/lib/kubelet/pods/9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9/volumes"
Oct 29 23:50:43.222818 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:43.222777    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"]
Oct 29 23:50:43.278670 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:43.278640    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:43.959892 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:43.959853    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn"]
Oct 29 23:50:44.072925 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:44.072872    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:44.186937 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:44.186909    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:44.317283 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:44.317205    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:45.551582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:45.551546    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:45.670401 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:45.670368    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:45.884752 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:45.884721    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:46.150707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:46.150618    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:47.367318 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:50:47.367284    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:50:48.195853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:48.195820    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:48.960324 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:48.960289    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn" podUID="8755d1b0-bf1b-45c5-8923-b2e448d967f7" containerName="payload-pre-processing" containerID="cri-o://fecffd2c39a1ebb251a915ef45ca30ff89b79f5c78ac4b286d7f82530b932f5b" gracePeriod=25
Oct 29 23:50:49.234920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:49.234896    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn"
Oct 29 23:50:49.295419 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:49.295383    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8755d1b0-bf1b-45c5-8923-b2e448d967f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8755d1b0-bf1b-45c5-8923-b2e448d967f7-plugins-config-volume\") pod \"8755d1b0-bf1b-45c5-8923-b2e448d967f7\" (UID: \"8755d1b0-bf1b-45c5-8923-b2e448d967f7\") "
Oct 29 23:50:49.295626 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:49.295525    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8755d1b0-bf1b-45c5-8923-b2e448d967f7-kube-api-access-hx4kj\" (UniqueName: \"kubernetes.io/projected/8755d1b0-bf1b-45c5-8923-b2e448d967f7-kube-api-access-hx4kj\") pod \"8755d1b0-bf1b-45c5-8923-b2e448d967f7\" (UID: \"8755d1b0-bf1b-45c5-8923-b2e448d967f7\") "
Oct 29 23:50:49.296221 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:49.296139    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8755d1b0-bf1b-45c5-8923-b2e448d967f7-plugins-config-volume" pod "8755d1b0-bf1b-45c5-8923-b2e448d967f7" (UID: "8755d1b0-bf1b-45c5-8923-b2e448d967f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:50:49.298288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:49.298257    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8755d1b0-bf1b-45c5-8923-b2e448d967f7-kube-api-access-hx4kj" pod "8755d1b0-bf1b-45c5-8923-b2e448d967f7" (UID: "8755d1b0-bf1b-45c5-8923-b2e448d967f7"). InnerVolumeSpecName "kube-api-access-hx4kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:50:49.397098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:49.397062    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8755d1b0-bf1b-45c5-8923-b2e448d967f7-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:49.397098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:49.397090    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx4kj\" (UniqueName: \"kubernetes.io/projected/8755d1b0-bf1b-45c5-8923-b2e448d967f7-kube-api-access-hx4kj\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:50.080539 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:50.080506    5428 generic.go:356] "Generic (PLEG): container finished" podID="8755d1b0-bf1b-45c5-8923-b2e448d967f7" containerID="fecffd2c39a1ebb251a915ef45ca30ff89b79f5c78ac4b286d7f82530b932f5b" exitCode=0
Oct 29 23:50:50.081029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:50.080563    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn"
Oct 29 23:50:50.081029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:50.080594    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn" event={"ID":"8755d1b0-bf1b-45c5-8923-b2e448d967f7","Type":"ContainerDied","Data":"fecffd2c39a1ebb251a915ef45ca30ff89b79f5c78ac4b286d7f82530b932f5b"}
Oct 29 23:50:50.081029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:50.080665    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn" event={"ID":"8755d1b0-bf1b-45c5-8923-b2e448d967f7","Type":"ContainerDied","Data":"b149aa26c0bce174a32f53720b4b6e9af1ccea14c27a2418eeb8d3a9b780093b"}
Oct 29 23:50:50.081029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:50.080687    5428 scope.go:122] "RemoveContainer" containerID="fecffd2c39a1ebb251a915ef45ca30ff89b79f5c78ac4b286d7f82530b932f5b"
Oct 29 23:50:50.092093 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:50.092076    5428 scope.go:122] "RemoveContainer" containerID="fecffd2c39a1ebb251a915ef45ca30ff89b79f5c78ac4b286d7f82530b932f5b"
Oct 29 23:50:50.092432 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:50:50.092406    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fecffd2c39a1ebb251a915ef45ca30ff89b79f5c78ac4b286d7f82530b932f5b\": container with ID starting with fecffd2c39a1ebb251a915ef45ca30ff89b79f5c78ac4b286d7f82530b932f5b not found: ID does not exist" containerID="fecffd2c39a1ebb251a915ef45ca30ff89b79f5c78ac4b286d7f82530b932f5b"
Oct 29 23:50:50.092552 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:50.092440    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fecffd2c39a1ebb251a915ef45ca30ff89b79f5c78ac4b286d7f82530b932f5b"} err="failed to get container status \"fecffd2c39a1ebb251a915ef45ca30ff89b79f5c78ac4b286d7f82530b932f5b\": rpc error: code = NotFound desc = could not find container \"fecffd2c39a1ebb251a915ef45ca30ff89b79f5c78ac4b286d7f82530b932f5b\": container with ID starting with fecffd2c39a1ebb251a915ef45ca30ff89b79f5c78ac4b286d7f82530b932f5b not found: ID does not exist"
Oct 29 23:50:50.132905 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:50.132876    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn"]
Oct 29 23:50:50.145757 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:50.145726    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2b0108-7f554ddbc-tbvmn"]
Oct 29 23:50:50.406576 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:50.406531    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:50.777197 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:50.777103    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8755d1b0-bf1b-45c5-8923-b2e448d967f7" path="/var/lib/kubelet/pods/8755d1b0-bf1b-45c5-8923-b2e448d967f7/volumes"
Oct 29 23:50:52.192486 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.192450    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn" podUID="22e5475e-1e3a-487e-a50c-97851f4e845d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:50:52.218613 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.218569    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:52.350792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.350763    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:52.465934 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.465832    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:52.855323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.855240    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"]
Oct 29 23:50:52.856001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.855979    5428 cpu_manager.go:414] "containerMap: removing container" podUID="4b8035b7-d41e-43bf-842e-9f9f9e54eae9" containerName="payload-pre-processing"
Oct 29 23:50:52.856128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856001    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8035b7-d41e-43bf-842e-9f9f9e54eae9" containerName="payload-pre-processing"
Oct 29 23:50:52.856128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856026    5428 cpu_manager.go:414] "containerMap: removing container" podUID="cf20849b-2d40-42ca-82fa-a91198bb012b" containerName="maas-api"
Oct 29 23:50:52.856128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856034    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf20849b-2d40-42ca-82fa-a91198bb012b" containerName="maas-api"
Oct 29 23:50:52.856128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856045    5428 cpu_manager.go:414] "containerMap: removing container" podUID="8fa0a50f-14ae-4ba8-a482-f7007672963c" containerName="payload-processing"
Oct 29 23:50:52.856128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856052    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa0a50f-14ae-4ba8-a482-f7007672963c" containerName="payload-processing"
Oct 29 23:50:52.856128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856063    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" containerName="istio-proxy"
Oct 29 23:50:52.856128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856071    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" containerName="istio-proxy"
Oct 29 23:50:52.856128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856082    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9df13b19-5a4b-4eff-8733-66c06fdb38eb" containerName="payload-processing"
Oct 29 23:50:52.856128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856089    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df13b19-5a4b-4eff-8733-66c06fdb38eb" containerName="payload-processing"
Oct 29 23:50:52.856128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856106    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b546f7f7-0c2c-4da9-a00d-41b52bb85815" containerName="maas-api"
Oct 29 23:50:52.856128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856113    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b546f7f7-0c2c-4da9-a00d-41b52bb85815" containerName="maas-api"
Oct 29 23:50:52.856128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856123    5428 cpu_manager.go:414] "containerMap: removing container" podUID="45b6bf03-44e3-43c8-9ed9-7984e56e031e" containerName="istio-proxy"
Oct 29 23:50:52.856128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856130    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45b6bf03-44e3-43c8-9ed9-7984e56e031e" containerName="istio-proxy"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856140    5428 cpu_manager.go:414] "containerMap: removing container" podUID="980d539c-9fe0-43d8-bff8-fcdb9aa963d9" containerName="maas-api"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856147    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="980d539c-9fe0-43d8-bff8-fcdb9aa963d9" containerName="maas-api"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856160    5428 cpu_manager.go:414] "containerMap: removing container" podUID="f083378a-6a46-4573-9c83-ec5af5a6e153" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856167    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f083378a-6a46-4573-9c83-ec5af5a6e153" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856185    5428 cpu_manager.go:414] "containerMap: removing container" podUID="f56ff730-ff0f-47f0-be77-3bbde2f85301" containerName="istio-proxy"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856192    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f56ff730-ff0f-47f0-be77-3bbde2f85301" containerName="istio-proxy"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856202    5428 cpu_manager.go:414] "containerMap: removing container" podUID="3d79a8de-da14-43a3-b4f1-cdb7fd66d235" containerName="payload-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856209    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d79a8de-da14-43a3-b4f1-cdb7fd66d235" containerName="payload-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856220    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9e14ce23-ae46-4db5-9275-ceaf9987ba61" containerName="maas-api"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856228    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e14ce23-ae46-4db5-9275-ceaf9987ba61" containerName="maas-api"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856240    5428 cpu_manager.go:414] "containerMap: removing container" podUID="103e3adb-1b76-44e2-ba11-03d11fea6b70" containerName="payload-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856247    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103e3adb-1b76-44e2-ba11-03d11fea6b70" containerName="payload-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856260    5428 cpu_manager.go:414] "containerMap: removing container" podUID="afe8000b-075b-4052-b38c-fe2a222b8306" containerName="maas-api"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856267    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe8000b-075b-4052-b38c-fe2a222b8306" containerName="maas-api"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856289    5428 cpu_manager.go:414] "containerMap: removing container" podUID="5180d7a3-b01d-4639-9421-f802b9155fa2" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856297    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5180d7a3-b01d-4639-9421-f802b9155fa2" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856310    5428 cpu_manager.go:414] "containerMap: removing container" podUID="5603bb2b-7fee-4663-9f2d-9a550a7cca59" containerName="maas-api"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856317    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5603bb2b-7fee-4663-9f2d-9a550a7cca59" containerName="maas-api"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856327    5428 cpu_manager.go:414] "containerMap: removing container" podUID="ad7987fc-c841-4693-94e3-36c855add12c" containerName="main"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856334    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad7987fc-c841-4693-94e3-36c855add12c" containerName="main"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856348    5428 cpu_manager.go:414] "containerMap: removing container" podUID="8755d1b0-bf1b-45c5-8923-b2e448d967f7" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856355    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8755d1b0-bf1b-45c5-8923-b2e448d967f7" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856374    5428 cpu_manager.go:414] "containerMap: removing container" podUID="85670e7f-43f1-4f46-b691-88beee106a99" containerName="payload-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856382    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85670e7f-43f1-4f46-b691-88beee106a99" containerName="payload-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856401    5428 cpu_manager.go:414] "containerMap: removing container" podUID="c1dae43c-236e-4510-a6b9-0ffa73efe69f" containerName="maas-api"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856407    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1dae43c-236e-4510-a6b9-0ffa73efe69f" containerName="maas-api"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856417    5428 cpu_manager.go:414] "containerMap: removing container" podUID="a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856424    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856436    5428 cpu_manager.go:414] "containerMap: removing container" podUID="92936ba4-7ef1-451e-9322-7cf3583600f5" containerName="maas-api"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856443    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92936ba4-7ef1-451e-9322-7cf3583600f5" containerName="maas-api"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856454    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e990be60-57d3-49f5-961a-d40381905773" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856461    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e990be60-57d3-49f5-961a-d40381905773" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856475    5428 cpu_manager.go:414] "containerMap: removing container" podUID="526c46bd-79b3-446f-8b63-7c341d98b102" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856482    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="526c46bd-79b3-446f-8b63-7c341d98b102" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856502    5428 cpu_manager.go:414] "containerMap: removing container" podUID="4c1689ec-01d8-4ef6-885c-e15f332d06a0" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856510    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1689ec-01d8-4ef6-885c-e15f332d06a0" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856521    5428 cpu_manager.go:414] "containerMap: removing container" podUID="3ab912fe-021d-406d-8120-c428d151f1d8" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856529    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ab912fe-021d-406d-8120-c428d151f1d8" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856541    5428 cpu_manager.go:414] "containerMap: removing container" podUID="6f08d29e-2d26-454f-9667-8c3b99f6855d" containerName="payload-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856548    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f08d29e-2d26-454f-9667-8c3b99f6855d" containerName="payload-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856569    5428 cpu_manager.go:414] "containerMap: removing container" podUID="8b10d05c-ccd8-4125-819e-19e71a233aa9" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856576    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b10d05c-ccd8-4125-819e-19e71a233aa9" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856588    5428 cpu_manager.go:414] "containerMap: removing container" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856596    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856608    5428 cpu_manager.go:414] "containerMap: removing container" podUID="3555bd1b-d45c-4196-953d-af8ed8af9e63" containerName="maas-api"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856615    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3555bd1b-d45c-4196-953d-af8ed8af9e63" containerName="maas-api"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856819    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1dae43c-236e-4510-a6b9-0ffa73efe69f" containerName="maas-api"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856834    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e8793d-377a-4e4c-a8d8-8ea2568b8cfc" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856847    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="92936ba4-7ef1-451e-9322-7cf3583600f5" containerName="maas-api"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856856    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e990be60-57d3-49f5-961a-d40381905773" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856866    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="526c46bd-79b3-446f-8b63-7c341d98b102" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856879    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c1689ec-01d8-4ef6-885c-e15f332d06a0" containerName="payload-pre-processing"
Oct 29 23:50:52.856863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856890    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ab912fe-021d-406d-8120-c428d151f1d8" containerName="payload-pre-processing"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856901    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f08d29e-2d26-454f-9667-8c3b99f6855d" containerName="payload-processing"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856913    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b10d05c-ccd8-4125-819e-19e71a233aa9" containerName="payload-pre-processing"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856922    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="90966e36-5c24-4642-97a3-dbe4825b6a10" containerName="main"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856929    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3555bd1b-d45c-4196-953d-af8ed8af9e63" containerName="maas-api"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856938    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8035b7-d41e-43bf-842e-9f9f9e54eae9" containerName="payload-pre-processing"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856949    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf20849b-2d40-42ca-82fa-a91198bb012b" containerName="maas-api"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856958    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa0a50f-14ae-4ba8-a482-f7007672963c" containerName="payload-processing"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856967    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cae6a36-4e41-4fda-a0e5-ceaa7a957cb9" containerName="istio-proxy"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856976    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df13b19-5a4b-4eff-8733-66c06fdb38eb" containerName="payload-processing"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856985    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b546f7f7-0c2c-4da9-a00d-41b52bb85815" containerName="maas-api"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.856994    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="45b6bf03-44e3-43c8-9ed9-7984e56e031e" containerName="istio-proxy"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.857004    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="980d539c-9fe0-43d8-bff8-fcdb9aa963d9" containerName="maas-api"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.857013    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f083378a-6a46-4573-9c83-ec5af5a6e153" containerName="payload-pre-processing"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.857023    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f56ff730-ff0f-47f0-be77-3bbde2f85301" containerName="istio-proxy"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.857034    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d79a8de-da14-43a3-b4f1-cdb7fd66d235" containerName="payload-processing"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.857041    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e14ce23-ae46-4db5-9275-ceaf9987ba61" containerName="maas-api"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.857051    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="103e3adb-1b76-44e2-ba11-03d11fea6b70" containerName="payload-processing"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.857059    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe8000b-075b-4052-b38c-fe2a222b8306" containerName="maas-api"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.857071    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5180d7a3-b01d-4639-9421-f802b9155fa2" containerName="payload-pre-processing"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.857080    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="5603bb2b-7fee-4663-9f2d-9a550a7cca59" containerName="maas-api"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.857089    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad7987fc-c841-4693-94e3-36c855add12c" containerName="main"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.857099    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8755d1b0-bf1b-45c5-8923-b2e448d967f7" containerName="payload-pre-processing"
Oct 29 23:50:52.859426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.857112    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="85670e7f-43f1-4f46-b691-88beee106a99" containerName="payload-processing"
Oct 29 23:50:52.862871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.862849    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:52.894932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.894902    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"]
Oct 29 23:50:52.943514 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.943486    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:52.943702 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.943565    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:52.943859 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.943842    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:52.943949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.943892    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:52.944008 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.943942    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bstb6\" (UniqueName: \"kubernetes.io/projected/29b17d6e-4c9f-4067-95b6-124bc1849a60-kube-api-access-bstb6\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:52.944078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.944065    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:52.944123 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.944093    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:52.944184 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.944166    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b17d6e-4c9f-4067-95b6-124bc1849a60-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:52.944233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.944219    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:52.959337 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:52.959315    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-2b0108-openshift-default-dockercfg-v8tck\""
Oct 29 23:50:53.045430 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.045400    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.045590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.045449    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.045663 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.045583    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bstb6\" (UniqueName: \"kubernetes.io/projected/29b17d6e-4c9f-4067-95b6-124bc1849a60-kube-api-access-bstb6\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.045725 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.045700    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.045783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.045738    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.045886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.045776    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.045886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.045829    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.045886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.045844    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b17d6e-4c9f-4067-95b6-124bc1849a60-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.045886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.045883    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.046120 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.045935    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.046120 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.046010    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.046120 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.046101    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.046405 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.046385    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.046518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.046500    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b17d6e-4c9f-4067-95b6-124bc1849a60-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.048152 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.048132    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.048235 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.048219    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.068410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.068387    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.068527 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.068450    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bstb6\" (UniqueName: \"kubernetes.io/projected/29b17d6e-4c9f-4067-95b6-124bc1849a60-kube-api-access-bstb6\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.196954 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.196924    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:53.532061 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.532036    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"]
Oct 29 23:50:53.533530 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:50:53.533507    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b17d6e_4c9f_4067_95b6_124bc1849a60.slice/crio-1ec8b1f789a96aa6cfd68c8faf69c21adc0b26d13af344714fa111b01b6efc86 WatchSource:0}: Error finding container 1ec8b1f789a96aa6cfd68c8faf69c21adc0b26d13af344714fa111b01b6efc86: Status 404 returned error can't find the container with id 1ec8b1f789a96aa6cfd68c8faf69c21adc0b26d13af344714fa111b01b6efc86
Oct 29 23:50:53.536383 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.536346    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:50:53.536502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.536451    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:50:53.536570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.536498    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:50:53.905187 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:53.905151    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:54.111087 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:54.111050    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz" event={"ID":"29b17d6e-4c9f-4067-95b6-124bc1849a60","Type":"ContainerStarted","Data":"cefba7c9427fa6ac5f4c49431654293361aacdf9bfad42aff77a19ded41d0c8c"}
Oct 29 23:50:54.111267 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:54.111111    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz" event={"ID":"29b17d6e-4c9f-4067-95b6-124bc1849a60","Type":"ContainerStarted","Data":"1ec8b1f789a96aa6cfd68c8faf69c21adc0b26d13af344714fa111b01b6efc86"}
Oct 29 23:50:54.171123 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:54.171013    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz" podStartSLOduration=2.170998902 podStartE2EDuration="2.170998902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:50:54.168037462 +0000 UTC m=+4057.510986208" watchObservedRunningTime="2026-10-29 23:50:54.170998902 +0000 UTC m=+4057.513947728"
Oct 29 23:50:54.197500 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:54.197474    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:50:54.198861 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:54.198835    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 29 23:50:54.198991 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:54.198882    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz" podUID="29b17d6e-4c9f-4067-95b6-124bc1849a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 29 23:50:55.197498 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:55.197455    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 29 23:50:55.197685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:55.197524    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz" podUID="29b17d6e-4c9f-4067-95b6-124bc1849a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 29 23:50:55.599783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:55.599747    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn"
Oct 29 23:50:55.702104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:55.702034    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22e5475e-1e3a-487e-a50c-97851f4e845d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e5475e-1e3a-487e-a50c-97851f4e845d-plugins-config-volume\") pod \"22e5475e-1e3a-487e-a50c-97851f4e845d\" (UID: \"22e5475e-1e3a-487e-a50c-97851f4e845d\") "
Oct 29 23:50:55.702104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:55.702076    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e5475e-1e3a-487e-a50c-97851f4e845d-kube-api-access-6p84c\" (UniqueName: \"kubernetes.io/projected/22e5475e-1e3a-487e-a50c-97851f4e845d-kube-api-access-6p84c\") pod \"22e5475e-1e3a-487e-a50c-97851f4e845d\" (UID: \"22e5475e-1e3a-487e-a50c-97851f4e845d\") "
Oct 29 23:50:55.702525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:55.702493    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e5475e-1e3a-487e-a50c-97851f4e845d-plugins-config-volume" pod "22e5475e-1e3a-487e-a50c-97851f4e845d" (UID: "22e5475e-1e3a-487e-a50c-97851f4e845d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:50:55.702736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:55.702713    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e5475e-1e3a-487e-a50c-97851f4e845d-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:55.704474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:55.704411    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e5475e-1e3a-487e-a50c-97851f4e845d-kube-api-access-6p84c" pod "22e5475e-1e3a-487e-a50c-97851f4e845d" (UID: "22e5475e-1e3a-487e-a50c-97851f4e845d"). InnerVolumeSpecName "kube-api-access-6p84c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:50:55.803972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:55.803935    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p84c\" (UniqueName: \"kubernetes.io/projected/22e5475e-1e3a-487e-a50c-97851f4e845d-kube-api-access-6p84c\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:55.885177 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:55.885141    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:56.126049 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:56.126019    5428 generic.go:356] "Generic (PLEG): container finished" podID="22e5475e-1e3a-487e-a50c-97851f4e845d" containerID="e396f1ea5cb5d84a2cf589e780a3a939cec7e7de97341e03df45a12b91c78bc0" exitCode=0
Oct 29 23:50:56.126239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:56.126070    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn"
Oct 29 23:50:56.126239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:56.126103    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn" event={"ID":"22e5475e-1e3a-487e-a50c-97851f4e845d","Type":"ContainerDied","Data":"e396f1ea5cb5d84a2cf589e780a3a939cec7e7de97341e03df45a12b91c78bc0"}
Oct 29 23:50:56.126239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:56.126146    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn" event={"ID":"22e5475e-1e3a-487e-a50c-97851f4e845d","Type":"ContainerDied","Data":"ada2ba827d8ffb759d4b6e693dd2b8b88e1dccdaf5698175247b01784dc70788"}
Oct 29 23:50:56.126239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:56.126170    5428 scope.go:122] "RemoveContainer" containerID="e396f1ea5cb5d84a2cf589e780a3a939cec7e7de97341e03df45a12b91c78bc0"
Oct 29 23:50:56.140413 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:56.140372    5428 scope.go:122] "RemoveContainer" containerID="e396f1ea5cb5d84a2cf589e780a3a939cec7e7de97341e03df45a12b91c78bc0"
Oct 29 23:50:56.140707 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:50:56.140683    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e396f1ea5cb5d84a2cf589e780a3a939cec7e7de97341e03df45a12b91c78bc0\": container with ID starting with e396f1ea5cb5d84a2cf589e780a3a939cec7e7de97341e03df45a12b91c78bc0 not found: ID does not exist" containerID="e396f1ea5cb5d84a2cf589e780a3a939cec7e7de97341e03df45a12b91c78bc0"
Oct 29 23:50:56.140810 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:56.140720    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e396f1ea5cb5d84a2cf589e780a3a939cec7e7de97341e03df45a12b91c78bc0"} err="failed to get container status \"e396f1ea5cb5d84a2cf589e780a3a939cec7e7de97341e03df45a12b91c78bc0\": rpc error: code = NotFound desc = could not find container \"e396f1ea5cb5d84a2cf589e780a3a939cec7e7de97341e03df45a12b91c78bc0\": container with ID starting with e396f1ea5cb5d84a2cf589e780a3a939cec7e7de97341e03df45a12b91c78bc0 not found: ID does not exist"
Oct 29 23:50:56.187915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:56.187850    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn"]
Oct 29 23:50:56.191025 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:56.190996    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-750972-6b984bdd88-b8xxn"]
Oct 29 23:50:56.197521 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:56.197498    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 29 23:50:56.197623 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:56.197539    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz" podUID="29b17d6e-4c9f-4067-95b6-124bc1849a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 29 23:50:56.791746 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:56.791709    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22e5475e-1e3a-487e-a50c-97851f4e845d" path="/var/lib/kubelet/pods/22e5475e-1e3a-487e-a50c-97851f4e845d/volumes"
Oct 29 23:50:56.888710 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:56.888675    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8"]
Oct 29 23:50:57.017350 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:57.017313    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:57.135270 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:57.135233    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:57.198071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:57.198037    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 29 23:50:57.198235 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:57.198096    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz" podUID="29b17d6e-4c9f-4067-95b6-124bc1849a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 29 23:50:58.198018 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.197933    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 29 23:50:58.198018 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.198005    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz" podUID="29b17d6e-4c9f-4067-95b6-124bc1849a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 29 23:50:58.243977 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.243945    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn" podUID="f232f882-a145-4350-9c1b-94340a32b926" containerName="main" containerID="cri-o://97e85a160ee7cb5ed3e7cf6fc3af79d7370c8b49c91c758752afc51a34976d2e" gracePeriod=45
Oct 29 23:50:58.258612 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.258587    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:50:58.593552 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.593520    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:50:58.643938 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.643905    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-tmp-dir\") pod \"f232f882-a145-4350-9c1b-94340a32b926\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") "
Oct 29 23:50:58.644129 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.644114    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-home\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-home\") pod \"f232f882-a145-4350-9c1b-94340a32b926\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") "
Oct 29 23:50:58.644239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.644219    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-model-cache\") pod \"f232f882-a145-4350-9c1b-94340a32b926\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") "
Oct 29 23:50:58.644361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.644258    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f232f882-a145-4350-9c1b-94340a32b926-kube-api-access-c7852\" (UniqueName: \"kubernetes.io/projected/f232f882-a145-4350-9c1b-94340a32b926-kube-api-access-c7852\") pod \"f232f882-a145-4350-9c1b-94340a32b926\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") "
Oct 29 23:50:58.644361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.644336    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f232f882-a145-4350-9c1b-94340a32b926-tls-certs\" (UniqueName: \"kubernetes.io/secret/f232f882-a145-4350-9c1b-94340a32b926-tls-certs\") pod \"f232f882-a145-4350-9c1b-94340a32b926\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") "
Oct 29 23:50:58.644523 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.644362    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-dshm\") pod \"f232f882-a145-4350-9c1b-94340a32b926\" (UID: \"f232f882-a145-4350-9c1b-94340a32b926\") "
Oct 29 23:50:58.646950 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.646916    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-tmp-dir" pod "f232f882-a145-4350-9c1b-94340a32b926" (UID: "f232f882-a145-4350-9c1b-94340a32b926"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:50:58.647210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.647160    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-home" pod "f232f882-a145-4350-9c1b-94340a32b926" (UID: "f232f882-a145-4350-9c1b-94340a32b926"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:50:58.647691 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.647667    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-model-cache" pod "f232f882-a145-4350-9c1b-94340a32b926" (UID: "f232f882-a145-4350-9c1b-94340a32b926"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:50:58.649449 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.649424    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f232f882-a145-4350-9c1b-94340a32b926-tls-certs" pod "f232f882-a145-4350-9c1b-94340a32b926" (UID: "f232f882-a145-4350-9c1b-94340a32b926"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:50:58.649984 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.649963    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-dshm" pod "f232f882-a145-4350-9c1b-94340a32b926" (UID: "f232f882-a145-4350-9c1b-94340a32b926"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:50:58.650171 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.650145    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f232f882-a145-4350-9c1b-94340a32b926-kube-api-access-c7852" pod "f232f882-a145-4350-9c1b-94340a32b926" (UID: "f232f882-a145-4350-9c1b-94340a32b926"). InnerVolumeSpecName "kube-api-access-c7852". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:50:58.745783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.745752    5428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-model-cache\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:58.745783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.745783    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7852\" (UniqueName: \"kubernetes.io/projected/f232f882-a145-4350-9c1b-94340a32b926-kube-api-access-c7852\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:58.746001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.745793    5428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f232f882-a145-4350-9c1b-94340a32b926-tls-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:58.746001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.745816    5428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-dshm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:58.746001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.745826    5428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-tmp-dir\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:58.746001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:58.745834    5428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f232f882-a145-4350-9c1b-94340a32b926-home\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:50:59.151669 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:59.151630    5428 generic.go:356] "Generic (PLEG): container finished" podID="f232f882-a145-4350-9c1b-94340a32b926" containerID="97e85a160ee7cb5ed3e7cf6fc3af79d7370c8b49c91c758752afc51a34976d2e" exitCode=0
Oct 29 23:50:59.151891 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:59.151674    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"
Oct 29 23:50:59.151891 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:59.151693    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn" event={"ID":"f232f882-a145-4350-9c1b-94340a32b926","Type":"ContainerDied","Data":"97e85a160ee7cb5ed3e7cf6fc3af79d7370c8b49c91c758752afc51a34976d2e"}
Oct 29 23:50:59.151891 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:59.151738    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn" event={"ID":"f232f882-a145-4350-9c1b-94340a32b926","Type":"ContainerDied","Data":"576b6cab180709779c35ec643eaf8a83cc1a98d10351956f2dde0ef0ddbf8431"}
Oct 29 23:50:59.151891 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:59.151757    5428 scope.go:122] "RemoveContainer" containerID="97e85a160ee7cb5ed3e7cf6fc3af79d7370c8b49c91c758752afc51a34976d2e"
Oct 29 23:50:59.186599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:59.186576    5428 scope.go:122] "RemoveContainer" containerID="97e85a160ee7cb5ed3e7cf6fc3af79d7370c8b49c91c758752afc51a34976d2e"
Oct 29 23:50:59.186982 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:50:59.186954    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e85a160ee7cb5ed3e7cf6fc3af79d7370c8b49c91c758752afc51a34976d2e\": container with ID starting with 97e85a160ee7cb5ed3e7cf6fc3af79d7370c8b49c91c758752afc51a34976d2e not found: ID does not exist" containerID="97e85a160ee7cb5ed3e7cf6fc3af79d7370c8b49c91c758752afc51a34976d2e"
Oct 29 23:50:59.187075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:59.186991    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e85a160ee7cb5ed3e7cf6fc3af79d7370c8b49c91c758752afc51a34976d2e"} err="failed to get container status \"97e85a160ee7cb5ed3e7cf6fc3af79d7370c8b49c91c758752afc51a34976d2e\": rpc error: code = NotFound desc = could not find container \"97e85a160ee7cb5ed3e7cf6fc3af79d7370c8b49c91c758752afc51a34976d2e\": container with ID starting with 97e85a160ee7cb5ed3e7cf6fc3af79d7370c8b49c91c758752afc51a34976d2e not found: ID does not exist"
Oct 29 23:50:59.197502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:59.197474    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 29 23:50:59.197639 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:59.197515    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz" podUID="29b17d6e-4c9f-4067-95b6-124bc1849a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 29 23:50:59.213436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:59.213409    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"]
Oct 29 23:50:59.225591 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:50:59.225567    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3b8149/test-model-3b8149-kserve-746d78c69c-cp5tn"]
Oct 29 23:51:00.198056 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:00.198021    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 29 23:51:00.198247 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:00.198074    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz" podUID="29b17d6e-4c9f-4067-95b6-124bc1849a60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 29 23:51:00.487429 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:00.487345    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:00.778315 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:00.778236    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f232f882-a145-4350-9c1b-94340a32b926" path="/var/lib/kubelet/pods/f232f882-a145-4350-9c1b-94340a32b926/volumes"
Oct 29 23:51:00.983610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:00.983570    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:01.202504 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:01.202476    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:51:01.202851 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:01.202830    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:51:01.203600 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:01.203582    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:51:01.889614 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:01.889585    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8" podUID="3f792ad2-06af-45dc-92a3-3e5514e9201a" containerName="payload-processing" containerID="cri-o://10e72c50b321c754dc6d460b78dab117d96e70792f0567a0b7954f3703e56a4e" gracePeriod=25
Oct 29 23:51:02.167233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:02.167199    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8"
Oct 29 23:51:02.191964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:02.191930    5428 generic.go:356] "Generic (PLEG): container finished" podID="3f792ad2-06af-45dc-92a3-3e5514e9201a" containerID="10e72c50b321c754dc6d460b78dab117d96e70792f0567a0b7954f3703e56a4e" exitCode=0
Oct 29 23:51:02.192132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:02.191971    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8"
Oct 29 23:51:02.192132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:02.192008    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8" event={"ID":"3f792ad2-06af-45dc-92a3-3e5514e9201a","Type":"ContainerDied","Data":"10e72c50b321c754dc6d460b78dab117d96e70792f0567a0b7954f3703e56a4e"}
Oct 29 23:51:02.192132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:02.192062    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8" event={"ID":"3f792ad2-06af-45dc-92a3-3e5514e9201a","Type":"ContainerDied","Data":"0ede949a20f997ad8466a11535ce8e01266a612205e7a2a3f52a9869ade2736d"}
Oct 29 23:51:02.192132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:02.192091    5428 scope.go:122] "RemoveContainer" containerID="10e72c50b321c754dc6d460b78dab117d96e70792f0567a0b7954f3703e56a4e"
Oct 29 23:51:02.205288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:02.205270    5428 scope.go:122] "RemoveContainer" containerID="10e72c50b321c754dc6d460b78dab117d96e70792f0567a0b7954f3703e56a4e"
Oct 29 23:51:02.205610 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:51:02.205578    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e72c50b321c754dc6d460b78dab117d96e70792f0567a0b7954f3703e56a4e\": container with ID starting with 10e72c50b321c754dc6d460b78dab117d96e70792f0567a0b7954f3703e56a4e not found: ID does not exist" containerID="10e72c50b321c754dc6d460b78dab117d96e70792f0567a0b7954f3703e56a4e"
Oct 29 23:51:02.205610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:02.205607    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e72c50b321c754dc6d460b78dab117d96e70792f0567a0b7954f3703e56a4e"} err="failed to get container status \"10e72c50b321c754dc6d460b78dab117d96e70792f0567a0b7954f3703e56a4e\": rpc error: code = NotFound desc = could not find container \"10e72c50b321c754dc6d460b78dab117d96e70792f0567a0b7954f3703e56a4e\": container with ID starting with 10e72c50b321c754dc6d460b78dab117d96e70792f0567a0b7954f3703e56a4e not found: ID does not exist"
Oct 29 23:51:02.212384 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:02.212362    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f792ad2-06af-45dc-92a3-3e5514e9201a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f792ad2-06af-45dc-92a3-3e5514e9201a-plugins-config-volume\") pod \"3f792ad2-06af-45dc-92a3-3e5514e9201a\" (UID: \"3f792ad2-06af-45dc-92a3-3e5514e9201a\") "
Oct 29 23:51:02.212520 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:02.212483    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f792ad2-06af-45dc-92a3-3e5514e9201a-kube-api-access-mgjjd\" (UniqueName: \"kubernetes.io/projected/3f792ad2-06af-45dc-92a3-3e5514e9201a-kube-api-access-mgjjd\") pod \"3f792ad2-06af-45dc-92a3-3e5514e9201a\" (UID: \"3f792ad2-06af-45dc-92a3-3e5514e9201a\") "
Oct 29 23:51:02.212735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:02.212713    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f792ad2-06af-45dc-92a3-3e5514e9201a-plugins-config-volume" pod "3f792ad2-06af-45dc-92a3-3e5514e9201a" (UID: "3f792ad2-06af-45dc-92a3-3e5514e9201a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:51:02.213229 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:02.213209    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f792ad2-06af-45dc-92a3-3e5514e9201a-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:02.215468 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:02.215447    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f792ad2-06af-45dc-92a3-3e5514e9201a-kube-api-access-mgjjd" pod "3f792ad2-06af-45dc-92a3-3e5514e9201a" (UID: "3f792ad2-06af-45dc-92a3-3e5514e9201a"). InnerVolumeSpecName "kube-api-access-mgjjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:51:02.314546 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:02.314497    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgjjd\" (UniqueName: \"kubernetes.io/projected/3f792ad2-06af-45dc-92a3-3e5514e9201a-kube-api-access-mgjjd\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:02.405747 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:02.405650    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:02.540504 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:02.540462    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8"]
Oct 29 23:51:02.546407 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:02.546379    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d20345-54bfc4d89d-nhgk8"]
Oct 29 23:51:02.778857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:02.778760    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f792ad2-06af-45dc-92a3-3e5514e9201a" path="/var/lib/kubelet/pods/3f792ad2-06af-45dc-92a3-3e5514e9201a/volumes"
Oct 29 23:51:04.402773 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:04.402733    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:04.756582 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:04.756497    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"]
Oct 29 23:51:04.804542 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:04.804499    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:06.301310 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:06.301268    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:08.136995 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:08.136962    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:09.911428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:09.911385    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:11.799880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:11.799836    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:13.590090 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:13.590054    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:15.271635 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:15.271584    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:15.570173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:15.570072    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:15.687121 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:15.687086    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:16.751415 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:16.751382    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"]
Oct 29 23:51:16.751939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:16.751587    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq" podUID="cb6ed96b-0059-4994-a0cd-8543d915a176" containerName="maas-api" containerID="cri-o://88638fc1ad71c5f201e44ae1c7b613728dd9fe6673dc8f0f4e41cc192d122f35" gracePeriod=30
Oct 29 23:51:16.829068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:16.829031    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:17.076606 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.076578    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:51:17.154568 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.154530    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb6ed96b-0059-4994-a0cd-8543d915a176-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6ed96b-0059-4994-a0cd-8543d915a176-maas-api-tls\") pod \"cb6ed96b-0059-4994-a0cd-8543d915a176\" (UID: \"cb6ed96b-0059-4994-a0cd-8543d915a176\") "
Oct 29 23:51:17.154793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.154653    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb6ed96b-0059-4994-a0cd-8543d915a176-kube-api-access-5wlvq\" (UniqueName: \"kubernetes.io/projected/cb6ed96b-0059-4994-a0cd-8543d915a176-kube-api-access-5wlvq\") pod \"cb6ed96b-0059-4994-a0cd-8543d915a176\" (UID: \"cb6ed96b-0059-4994-a0cd-8543d915a176\") "
Oct 29 23:51:17.154793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.154728    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb6ed96b-0059-4994-a0cd-8543d915a176-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6ed96b-0059-4994-a0cd-8543d915a176-metrics-certs\") pod \"cb6ed96b-0059-4994-a0cd-8543d915a176\" (UID: \"cb6ed96b-0059-4994-a0cd-8543d915a176\") "
Oct 29 23:51:17.154938 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.154916    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb6ed96b-0059-4994-a0cd-8543d915a176-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6ed96b-0059-4994-a0cd-8543d915a176-openshift-service-ca\") pod \"cb6ed96b-0059-4994-a0cd-8543d915a176\" (UID: \"cb6ed96b-0059-4994-a0cd-8543d915a176\") "
Oct 29 23:51:17.156130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.156002    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb6ed96b-0059-4994-a0cd-8543d915a176-openshift-service-ca" pod "cb6ed96b-0059-4994-a0cd-8543d915a176" (UID: "cb6ed96b-0059-4994-a0cd-8543d915a176"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:51:17.158357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.158303    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb6ed96b-0059-4994-a0cd-8543d915a176-metrics-certs" pod "cb6ed96b-0059-4994-a0cd-8543d915a176" (UID: "cb6ed96b-0059-4994-a0cd-8543d915a176"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:51:17.159898 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.159850    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6ed96b-0059-4994-a0cd-8543d915a176-kube-api-access-5wlvq" pod "cb6ed96b-0059-4994-a0cd-8543d915a176" (UID: "cb6ed96b-0059-4994-a0cd-8543d915a176"). InnerVolumeSpecName "kube-api-access-5wlvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:51:17.161410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.161124    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb6ed96b-0059-4994-a0cd-8543d915a176-maas-api-tls" pod "cb6ed96b-0059-4994-a0cd-8543d915a176" (UID: "cb6ed96b-0059-4994-a0cd-8543d915a176"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:51:17.257011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.256964    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6ed96b-0059-4994-a0cd-8543d915a176-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:17.257011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.257004    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6ed96b-0059-4994-a0cd-8543d915a176-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:17.257278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.257020    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wlvq\" (UniqueName: \"kubernetes.io/projected/cb6ed96b-0059-4994-a0cd-8543d915a176-kube-api-access-5wlvq\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:17.257278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.257036    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6ed96b-0059-4994-a0cd-8543d915a176-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:17.370739 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:51:17.370703    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:51:17.374873 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.374845    5428 generic.go:356] "Generic (PLEG): container finished" podID="cb6ed96b-0059-4994-a0cd-8543d915a176" containerID="88638fc1ad71c5f201e44ae1c7b613728dd9fe6673dc8f0f4e41cc192d122f35" exitCode=0
Oct 29 23:51:17.375016 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.374903    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"
Oct 29 23:51:17.375016 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.374940    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq" event={"ID":"cb6ed96b-0059-4994-a0cd-8543d915a176","Type":"ContainerDied","Data":"88638fc1ad71c5f201e44ae1c7b613728dd9fe6673dc8f0f4e41cc192d122f35"}
Oct 29 23:51:17.375016 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.375004    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq" event={"ID":"cb6ed96b-0059-4994-a0cd-8543d915a176","Type":"ContainerDied","Data":"b95746d0a24cf2955a8290e369049f75e00c1ff86cc7905e2b7e2ea887924502"}
Oct 29 23:51:17.375213 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.375036    5428 scope.go:122] "RemoveContainer" containerID="88638fc1ad71c5f201e44ae1c7b613728dd9fe6673dc8f0f4e41cc192d122f35"
Oct 29 23:51:17.383602 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.383446    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:17.387392 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.387373    5428 scope.go:122] "RemoveContainer" containerID="88638fc1ad71c5f201e44ae1c7b613728dd9fe6673dc8f0f4e41cc192d122f35"
Oct 29 23:51:17.387702 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:51:17.387684    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88638fc1ad71c5f201e44ae1c7b613728dd9fe6673dc8f0f4e41cc192d122f35\": container with ID starting with 88638fc1ad71c5f201e44ae1c7b613728dd9fe6673dc8f0f4e41cc192d122f35 not found: ID does not exist" containerID="88638fc1ad71c5f201e44ae1c7b613728dd9fe6673dc8f0f4e41cc192d122f35"
Oct 29 23:51:17.387764 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.387710    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88638fc1ad71c5f201e44ae1c7b613728dd9fe6673dc8f0f4e41cc192d122f35"} err="failed to get container status \"88638fc1ad71c5f201e44ae1c7b613728dd9fe6673dc8f0f4e41cc192d122f35\": rpc error: code = NotFound desc = could not find container \"88638fc1ad71c5f201e44ae1c7b613728dd9fe6673dc8f0f4e41cc192d122f35\": container with ID starting with 88638fc1ad71c5f201e44ae1c7b613728dd9fe6673dc8f0f4e41cc192d122f35 not found: ID does not exist"
Oct 29 23:51:17.431841 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.431796    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"]
Oct 29 23:51:17.438130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.438105    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-2b0108-8596cd8774-cmbcq"]
Oct 29 23:51:17.555574 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.555540    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:17.683010 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:17.682920    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:18.778142 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:18.778110    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb6ed96b-0059-4994-a0cd-8543d915a176" path="/var/lib/kubelet/pods/cb6ed96b-0059-4994-a0cd-8543d915a176/volumes"
Oct 29 23:51:19.775337 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:19.775303    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:19.782230 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:19.782193    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj" podUID="d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a" containerName="main" containerID="cri-o://f060cb872bf111e1b3738d92bceec2cd5679aadc6f58d5830a8c260495a47dc2" gracePeriod=45
Oct 29 23:51:19.943880 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:19.943838    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj" podUID="d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a" containerName="main" probeResult="failure" output="Get \"https://10.133.1.27:8000/ready\": dial tcp 10.133.1.27:8000: connect: connection refused"
Oct 29 23:51:20.043315 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.043259    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:51:20.115778 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.115751    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-kube-api-access-djh9x\" (UniqueName: \"kubernetes.io/projected/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-kube-api-access-djh9x\") pod \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") "
Oct 29 23:51:20.115996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.115910    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-model-cache\") pod \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") "
Oct 29 23:51:20.115996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.115992    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-dshm\") pod \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") "
Oct 29 23:51:20.116132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.116018    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-home\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-home\") pod \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") "
Oct 29 23:51:20.116132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.116046    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-tmp-dir\") pod \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") "
Oct 29 23:51:20.116132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.116073    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-tls-certs\") pod \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\" (UID: \"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a\") "
Oct 29 23:51:20.116298 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.116221    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-model-cache" pod "d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a" (UID: "d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:20.116349 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.116302    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-home" pod "d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a" (UID: "d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:20.116465 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.116439    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-tmp-dir" pod "d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a" (UID: "d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:20.116730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.116702    5428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-home\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:20.116730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.116727    5428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-tmp-dir\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:20.116932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.116742    5428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-model-cache\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:20.118413 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.118381    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-dshm" pod "d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a" (UID: "d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:20.118518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.118405    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-tls-certs" pod "d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a" (UID: "d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:51:20.118518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.118431    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-kube-api-access-djh9x" pod "d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a" (UID: "d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a"). InnerVolumeSpecName "kube-api-access-djh9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:51:20.217889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.217859    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djh9x\" (UniqueName: \"kubernetes.io/projected/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-kube-api-access-djh9x\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:20.217889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.217884    5428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-dshm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:20.217889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.217895    5428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a-tls-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:20.398696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.398658    5428 generic.go:356] "Generic (PLEG): container finished" podID="d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a" containerID="f060cb872bf111e1b3738d92bceec2cd5679aadc6f58d5830a8c260495a47dc2" exitCode=0
Oct 29 23:51:20.398896 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.398700    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"
Oct 29 23:51:20.398896 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.398752    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj" event={"ID":"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a","Type":"ContainerDied","Data":"f060cb872bf111e1b3738d92bceec2cd5679aadc6f58d5830a8c260495a47dc2"}
Oct 29 23:51:20.398896 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.398793    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj" event={"ID":"d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a","Type":"ContainerDied","Data":"037414b7e4e775a2b5aaa8c5fe198c9cd6e9d4c48a9ef627efda5aabf86045c1"}
Oct 29 23:51:20.398896 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.398826    5428 scope.go:122] "RemoveContainer" containerID="f060cb872bf111e1b3738d92bceec2cd5679aadc6f58d5830a8c260495a47dc2"
Oct 29 23:51:20.410585 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.410564    5428 scope.go:122] "RemoveContainer" containerID="f060cb872bf111e1b3738d92bceec2cd5679aadc6f58d5830a8c260495a47dc2"
Oct 29 23:51:20.410895 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:51:20.410875    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f060cb872bf111e1b3738d92bceec2cd5679aadc6f58d5830a8c260495a47dc2\": container with ID starting with f060cb872bf111e1b3738d92bceec2cd5679aadc6f58d5830a8c260495a47dc2 not found: ID does not exist" containerID="f060cb872bf111e1b3738d92bceec2cd5679aadc6f58d5830a8c260495a47dc2"
Oct 29 23:51:20.410971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.410901    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f060cb872bf111e1b3738d92bceec2cd5679aadc6f58d5830a8c260495a47dc2"} err="failed to get container status \"f060cb872bf111e1b3738d92bceec2cd5679aadc6f58d5830a8c260495a47dc2\": rpc error: code = NotFound desc = could not find container \"f060cb872bf111e1b3738d92bceec2cd5679aadc6f58d5830a8c260495a47dc2\": container with ID starting with f060cb872bf111e1b3738d92bceec2cd5679aadc6f58d5830a8c260495a47dc2 not found: ID does not exist"
Oct 29 23:51:20.451054 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.451020    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"]
Oct 29 23:51:20.454656 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.454632    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-825472/test-model-825472-kserve-767979fd87-dghdj"]
Oct 29 23:51:20.778129 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:20.778052    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a" path="/var/lib/kubelet/pods/d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a/volumes"
Oct 29 23:51:21.536079 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:21.536048    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:22.517715 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.517683    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:22.957071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.957038    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"]
Oct 29 23:51:22.957702 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.957678    5428 cpu_manager.go:414] "containerMap: removing container" podUID="22e5475e-1e3a-487e-a50c-97851f4e845d" containerName="payload-processing"
Oct 29 23:51:22.957702 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.957693    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e5475e-1e3a-487e-a50c-97851f4e845d" containerName="payload-processing"
Oct 29 23:51:22.957871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.957711    5428 cpu_manager.go:414] "containerMap: removing container" podUID="d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a" containerName="main"
Oct 29 23:51:22.957871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.957717    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a" containerName="main"
Oct 29 23:51:22.957871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.957752    5428 cpu_manager.go:414] "containerMap: removing container" podUID="cb6ed96b-0059-4994-a0cd-8543d915a176" containerName="maas-api"
Oct 29 23:51:22.957871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.957759    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb6ed96b-0059-4994-a0cd-8543d915a176" containerName="maas-api"
Oct 29 23:51:22.957871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.957777    5428 cpu_manager.go:414] "containerMap: removing container" podUID="f232f882-a145-4350-9c1b-94340a32b926" containerName="main"
Oct 29 23:51:22.957871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.957782    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f232f882-a145-4350-9c1b-94340a32b926" containerName="main"
Oct 29 23:51:22.957871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.957789    5428 cpu_manager.go:414] "containerMap: removing container" podUID="3f792ad2-06af-45dc-92a3-3e5514e9201a" containerName="payload-processing"
Oct 29 23:51:22.957871 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.957793    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f792ad2-06af-45dc-92a3-3e5514e9201a" containerName="payload-processing"
Oct 29 23:51:22.958252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.957917    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb6ed96b-0059-4994-a0cd-8543d915a176" containerName="maas-api"
Oct 29 23:51:22.958252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.957931    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="f232f882-a145-4350-9c1b-94340a32b926" containerName="main"
Oct 29 23:51:22.958252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.957938    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f792ad2-06af-45dc-92a3-3e5514e9201a" containerName="payload-processing"
Oct 29 23:51:22.958252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.957945    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e5475e-1e3a-487e-a50c-97851f4e845d" containerName="payload-processing"
Oct 29 23:51:22.958252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.957957    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5fbdcde-d3a1-4ebb-8f9b-b1ed8fcfa47a" containerName="main"
Oct 29 23:51:22.964936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.964914    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:22.987782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:22.987746    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"]
Oct 29 23:51:23.001055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.001015    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"]
Oct 29 23:51:23.007102 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.007072    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.024686 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.024657    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"]
Oct 29 23:51:23.055825 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.055772    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.056027 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.055861    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.056027 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.055914    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.056027 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.055949    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.056027 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.055998    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8dxb\" (UniqueName: \"kubernetes.io/projected/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-kube-api-access-w8dxb\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.056252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.056069    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.056252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.056095    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.056252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.056161    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.056252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.056207    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.056252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.056230    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.056570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.056275    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.056570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.056297    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.056570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.056347    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.056570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.056371    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.056570 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.056542    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.057599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.056621    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.057599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.056644    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz825\" (UniqueName: \"kubernetes.io/projected/0d2f66a5-0f31-4820-9693-b4d87b6c4933-kube-api-access-jz825\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.057599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.056671    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.157667 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.157617    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.157906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.157737    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz825\" (UniqueName: \"kubernetes.io/projected/0d2f66a5-0f31-4820-9693-b4d87b6c4933-kube-api-access-jz825\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.157906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.157772    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.157906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.157820    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.157906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.157850    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.158408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.157910    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.158408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.157950    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.158408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.158041    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8dxb\" (UniqueName: \"kubernetes.io/projected/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-kube-api-access-w8dxb\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.158408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.158129    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.158408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.158159    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.158408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.158237    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.158408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.158276    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.158408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.158297    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.158408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.158323    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.158408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.158352    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.158408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.158379    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.159052 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.158546    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.159052 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.158583    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.159052 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.158617    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.159052 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.158637    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.159052 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.158662    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.159052 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.158750    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-credential-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.159052 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.158775    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.159052 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.158778    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.159895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.159071    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-workload-socket\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.159895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.159101    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istiod-ca-cert\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.159895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.159260    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-data\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.159895 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.159311    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-workload-certs\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.161347 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.161325    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.161485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.161393    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.161546 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.161484    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-envoy\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.161648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.161627    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-podinfo\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.181029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.181002    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.181136 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.181050    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8dxb\" (UniqueName: \"kubernetes.io/projected/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-kube-api-access-w8dxb\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.181200 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.181178    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz825\" (UniqueName: \"kubernetes.io/projected/0d2f66a5-0f31-4820-9693-b4d87b6c4933-kube-api-access-jz825\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.181789 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.181770    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-token\") pod \"e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") " pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.284209 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.284139    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:23.329926 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.329901    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:23.394374 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.394320    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:23.618566 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.618535    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"]
Oct 29 23:51:23.619816 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:51:23.619780    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2f66a5_0f31_4820_9693_b4d87b6c4933.slice/crio-d5a7fdaaf4aeff7b39db340bb64bcd786b4060ca2c64b25d4980156625448e8f WatchSource:0}: Error finding container d5a7fdaaf4aeff7b39db340bb64bcd786b4060ca2c64b25d4980156625448e8f: Status 404 returned error can't find the container with id d5a7fdaaf4aeff7b39db340bb64bcd786b4060ca2c64b25d4980156625448e8f
Oct 29 23:51:23.622600 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.622566    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:51:23.622691 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.622634    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:51:23.622691 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.622660    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:51:23.664471 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.664443    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"]
Oct 29 23:51:23.665685 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:51:23.665663    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4d02b4_5d2b_466b_a379_fcbdd26b8f32.slice/crio-54044d46db47312e3a64cd758c9f59e519ac324b50b6404d1b1b515f1466b027 WatchSource:0}: Error finding container 54044d46db47312e3a64cd758c9f59e519ac324b50b6404d1b1b515f1466b027: Status 404 returned error can't find the container with id 54044d46db47312e3a64cd758c9f59e519ac324b50b6404d1b1b515f1466b027
Oct 29 23:51:23.668270 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.668240    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:51:23.668360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.668321    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:51:23.668360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:23.668351    5428 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 23:51:24.454973 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:24.454932    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v" event={"ID":"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32","Type":"ContainerStarted","Data":"a958305d467bc69cc3b61cd2dd63e6297fa4d8db5a914e150b3c06c4783e0d9e"}
Oct 29 23:51:24.455455 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:24.454996    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v" event={"ID":"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32","Type":"ContainerStarted","Data":"54044d46db47312e3a64cd758c9f59e519ac324b50b6404d1b1b515f1466b027"}
Oct 29 23:51:24.456693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:24.456666    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg" event={"ID":"0d2f66a5-0f31-4820-9693-b4d87b6c4933","Type":"ContainerStarted","Data":"010f55a43d14481a8b9c383ed944b5fa15e3b9c4b6a62cc203439617ac7a2ab6"}
Oct 29 23:51:24.456867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:24.456716    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg" event={"ID":"0d2f66a5-0f31-4820-9693-b4d87b6c4933","Type":"ContainerStarted","Data":"d5a7fdaaf4aeff7b39db340bb64bcd786b4060ca2c64b25d4980156625448e8f"}
Oct 29 23:51:24.560826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:24.558320    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v" podStartSLOduration=2.5583001210000003 podStartE2EDuration="2.558300121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:51:24.513524518 +0000 UTC m=+4087.856473269" watchObservedRunningTime="2026-10-29 23:51:24.558300121 +0000 UTC m=+4087.901248867"
Oct 29 23:51:24.613024 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:24.612949    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg" podStartSLOduration=2.612927317 podStartE2EDuration="2.612927317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:51:24.557337875 +0000 UTC m=+4087.900286622" watchObservedRunningTime="2026-10-29 23:51:24.612927317 +0000 UTC m=+4087.955876067"
Oct 29 23:51:25.235496 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:25.235467    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:25.286826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:25.286428    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:25.303826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:25.298794    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Oct 29 23:51:25.303826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:25.298877    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg" podUID="0d2f66a5-0f31-4820-9693-b4d87b6c4933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Oct 29 23:51:25.331039 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:25.331002    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:25.332544 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:25.332518    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 29 23:51:25.332632 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:25.332568    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v" podUID="8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 29 23:51:26.285822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:26.284998    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Oct 29 23:51:26.285822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:26.285061    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg" podUID="0d2f66a5-0f31-4820-9693-b4d87b6c4933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Oct 29 23:51:26.331003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:26.330971    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 29 23:51:26.331172 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:26.331015    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v" podUID="8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 29 23:51:27.209827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:27.194248    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:27.291827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:27.288396    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Oct 29 23:51:27.291827 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:27.288457    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg" podUID="0d2f66a5-0f31-4820-9693-b4d87b6c4933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Oct 29 23:51:27.330279 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:27.330248    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 29 23:51:27.330457 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:27.330293    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v" podUID="8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 29 23:51:28.091636 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:28.091593    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:28.285824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:28.285500    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Oct 29 23:51:28.285824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:28.285560    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg" podUID="0d2f66a5-0f31-4820-9693-b4d87b6c4933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Oct 29 23:51:28.330403 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:28.330371    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 29 23:51:28.330786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:28.330419    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v" podUID="8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 29 23:51:29.126710 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:29.126678    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:29.294987 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:29.288173    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Oct 29 23:51:29.294987 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:29.288235    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg" podUID="0d2f66a5-0f31-4820-9693-b4d87b6c4933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Oct 29 23:51:29.330684 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:29.330650    5428 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Oct 29 23:51:29.331113 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:29.330700    5428 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v" podUID="8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Oct 29 23:51:30.289415 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:30.289386    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:30.289853 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:30.289832    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:30.290487 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:30.290466    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:30.335382 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:30.335351    5428 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:30.335737 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:30.335623    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:30.336393 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:30.336374    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:31.065203 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:31.065170    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:32.558786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:32.558744    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:33.228190 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:33.228158    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:33.352621 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:33.352588    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:33.467313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:33.467274    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:33.866789 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:33.866750    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:33.977201 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:33.977170    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:34.104761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:34.104729    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:34.841509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:34.841464    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:34.953095 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:34.953064    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:35.192346 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:35.192301    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"]
Oct 29 23:51:35.194152 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:35.194107    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v" podUID="8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" containerName="istio-proxy" containerID="cri-o://a958305d467bc69cc3b61cd2dd63e6297fa4d8db5a914e150b3c06c4783e0d9e" gracePeriod=30
Oct 29 23:51:35.214192 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:35.213141    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"]
Oct 29 23:51:35.214192 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:35.213443    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg" podUID="0d2f66a5-0f31-4820-9693-b4d87b6c4933" containerName="istio-proxy" containerID="cri-o://010f55a43d14481a8b9c383ed944b5fa15e3b9c4b6a62cc203439617ac7a2ab6" gracePeriod=30
Oct 29 23:51:35.226490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:35.226460    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:35.230301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:35.230273    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"]
Oct 29 23:51:35.230550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:35.230527    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz" podUID="29b17d6e-4c9f-4067-95b6-124bc1849a60" containerName="istio-proxy" containerID="cri-o://cefba7c9427fa6ac5f4c49431654293361aacdf9bfad42aff77a19ded41d0c8c" gracePeriod=30
Oct 29 23:51:35.379230 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:35.378898    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:35.731471 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:35.731435    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:37.396096 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:37.396062    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:39.415572 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:39.415539    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:39.928915 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:39.928878    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"]
Oct 29 23:51:39.976339 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:39.976302    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:40.314512 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.314478    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:40.515860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.515758    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:40.575575 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.574400    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:40.597080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.596855    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"]
Oct 29 23:51:40.597247 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.597152    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr" podUID="97ec11c4-83fa-4ae5-94fa-35a14eed785f" containerName="istio-proxy" containerID="cri-o://6d28c6b1a570b2d39a78079bc2eb077dbd8fa383f6c04ef7323de76b04d9e893" gracePeriod=30
Oct 29 23:51:40.612875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.611525    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.633697    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-podinfo\") pod \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") "
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.633748    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-workload-socket\") pod \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") "
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.633783    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-credential-socket\") pod \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") "
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.633893    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-token\" (UniqueName: \"kubernetes.io/projected/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-token\") pod \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") "
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.633934    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-podinfo\") pod \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") "
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.633981    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-envoy\") pod \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") "
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.634010    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-kube-api-access-w8dxb\" (UniqueName: \"kubernetes.io/projected/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-kube-api-access-w8dxb\") pod \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") "
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.634076    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istiod-ca-cert\") pod \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") "
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.634105    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-data\") pod \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") "
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.634137    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-workload-socket\") pod \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") "
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.634257    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-workload-certs\") pod \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") "
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.634301    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-envoy\") pod \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") "
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.634336    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2f66a5-0f31-4820-9693-b4d87b6c4933-kube-api-access-jz825\" (UniqueName: \"kubernetes.io/projected/0d2f66a5-0f31-4820-9693-b4d87b6c4933-kube-api-access-jz825\") pod \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") "
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.634381    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istiod-ca-cert\") pod \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") "
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.634444    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-workload-certs\") pod \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") "
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.634480    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-data\") pod \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") "
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.634524    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-token\" (UniqueName: \"kubernetes.io/projected/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-token\") pod \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\" (UID: \"0d2f66a5-0f31-4820-9693-b4d87b6c4933\") "
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.634567    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-credential-socket\") pod \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\" (UID: \"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32\") "
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.636654    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-workload-socket" pod "0d2f66a5-0f31-4820-9693-b4d87b6c4933" (UID: "0d2f66a5-0f31-4820-9693-b4d87b6c4933"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.638892    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-data" pod "0d2f66a5-0f31-4820-9693-b4d87b6c4933" (UID: "0d2f66a5-0f31-4820-9693-b4d87b6c4933"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.639831    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-credential-socket" pod "0d2f66a5-0f31-4820-9693-b4d87b6c4933" (UID: "0d2f66a5-0f31-4820-9693-b4d87b6c4933"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.644907    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-workload-certs" pod "0d2f66a5-0f31-4820-9693-b4d87b6c4933" (UID: "0d2f66a5-0f31-4820-9693-b4d87b6c4933"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.644963    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.645164    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-data" pod "8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" (UID: "8c4d02b4-5d2b-466b-a379-fcbdd26b8f32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.646247    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istiod-ca-cert" pod "8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" (UID: "8c4d02b4-5d2b-466b-a379-fcbdd26b8f32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.646456    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-workload-certs" pod "8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" (UID: "8c4d02b4-5d2b-466b-a379-fcbdd26b8f32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:40.647158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.646692    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-workload-socket" pod "8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" (UID: "8c4d02b4-5d2b-466b-a379-fcbdd26b8f32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:40.654260 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.648967    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istiod-ca-cert" pod "0d2f66a5-0f31-4820-9693-b4d87b6c4933" (UID: "0d2f66a5-0f31-4820-9693-b4d87b6c4933"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:51:40.654260 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.649035    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-credential-socket" pod "8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" (UID: "8c4d02b4-5d2b-466b-a379-fcbdd26b8f32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:40.654260 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.649153    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-token" pod "0d2f66a5-0f31-4820-9693-b4d87b6c4933" (UID: "0d2f66a5-0f31-4820-9693-b4d87b6c4933"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:51:40.654260 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.651565    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-envoy" pod "0d2f66a5-0f31-4820-9693-b4d87b6c4933" (UID: "0d2f66a5-0f31-4820-9693-b4d87b6c4933"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:40.654260 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.654087    5428 generic.go:356] "Generic (PLEG): container finished" podID="29b17d6e-4c9f-4067-95b6-124bc1849a60" containerID="cefba7c9427fa6ac5f4c49431654293361aacdf9bfad42aff77a19ded41d0c8c" exitCode=0
Oct 29 23:51:40.654260 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.654162    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz" event={"ID":"29b17d6e-4c9f-4067-95b6-124bc1849a60","Type":"ContainerDied","Data":"cefba7c9427fa6ac5f4c49431654293361aacdf9bfad42aff77a19ded41d0c8c"}
Oct 29 23:51:40.671410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.670231    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-podinfo" pod "0d2f66a5-0f31-4820-9693-b4d87b6c4933" (UID: "0d2f66a5-0f31-4820-9693-b4d87b6c4933"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:51:40.671410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.670354    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-kube-api-access-w8dxb" pod "8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" (UID: "8c4d02b4-5d2b-466b-a379-fcbdd26b8f32"). InnerVolumeSpecName "kube-api-access-w8dxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:51:40.671410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.670419    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-envoy" pod "8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" (UID: "8c4d02b4-5d2b-466b-a379-fcbdd26b8f32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:40.671410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.671357    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-token" pod "8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" (UID: "8c4d02b4-5d2b-466b-a379-fcbdd26b8f32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:51:40.672669 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.672637    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2f66a5-0f31-4820-9693-b4d87b6c4933-kube-api-access-jz825" pod "0d2f66a5-0f31-4820-9693-b4d87b6c4933" (UID: "0d2f66a5-0f31-4820-9693-b4d87b6c4933"). InnerVolumeSpecName "kube-api-access-jz825". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:51:40.674154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.674089    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-podinfo" pod "8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" (UID: "8c4d02b4-5d2b-466b-a379-fcbdd26b8f32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.766095    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.766137    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.766150    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.766162    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.766175    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.766191    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.766205    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.766216    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.766240    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.766255    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.766268    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8dxb\" (UniqueName: \"kubernetes.io/projected/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-kube-api-access-w8dxb\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.766280    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.766299    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.766310    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.766322    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.766334    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d2f66a5-0f31-4820-9693-b4d87b6c4933-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.766349    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz825\" (UniqueName: \"kubernetes.io/projected/0d2f66a5-0f31-4820-9693-b4d87b6c4933-kube-api-access-jz825\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.766362    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.767515    5428 generic.go:356] "Generic (PLEG): container finished" podID="0d2f66a5-0f31-4820-9693-b4d87b6c4933" containerID="010f55a43d14481a8b9c383ed944b5fa15e3b9c4b6a62cc203439617ac7a2ab6" exitCode=0
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.767697    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg" event={"ID":"0d2f66a5-0f31-4820-9693-b4d87b6c4933","Type":"ContainerDied","Data":"010f55a43d14481a8b9c383ed944b5fa15e3b9c4b6a62cc203439617ac7a2ab6"}
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.767864    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg" event={"ID":"0d2f66a5-0f31-4820-9693-b4d87b6c4933","Type":"ContainerDied","Data":"d5a7fdaaf4aeff7b39db340bb64bcd786b4060ca2c64b25d4980156625448e8f"}
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.767912    5428 scope.go:122] "RemoveContainer" containerID="010f55a43d14481a8b9c383ed944b5fa15e3b9c4b6a62cc203439617ac7a2ab6"
Oct 29 23:51:40.772243 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.768111    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"
Oct 29 23:51:40.790174 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.783564    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:51:40.795303 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.795260    5428 generic.go:356] "Generic (PLEG): container finished" podID="8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" containerID="a958305d467bc69cc3b61cd2dd63e6297fa4d8db5a914e150b3c06c4783e0d9e" exitCode=0
Oct 29 23:51:40.796055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.795597    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"
Oct 29 23:51:40.805115 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.805070    5428 scope.go:122] "RemoveContainer" containerID="010f55a43d14481a8b9c383ed944b5fa15e3b9c4b6a62cc203439617ac7a2ab6"
Oct 29 23:51:40.808589 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:51:40.808548    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"010f55a43d14481a8b9c383ed944b5fa15e3b9c4b6a62cc203439617ac7a2ab6\": container with ID starting with 010f55a43d14481a8b9c383ed944b5fa15e3b9c4b6a62cc203439617ac7a2ab6 not found: ID does not exist" containerID="010f55a43d14481a8b9c383ed944b5fa15e3b9c4b6a62cc203439617ac7a2ab6"
Oct 29 23:51:40.808681 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.808600    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010f55a43d14481a8b9c383ed944b5fa15e3b9c4b6a62cc203439617ac7a2ab6"} err="failed to get container status \"010f55a43d14481a8b9c383ed944b5fa15e3b9c4b6a62cc203439617ac7a2ab6\": rpc error: code = NotFound desc = could not find container \"010f55a43d14481a8b9c383ed944b5fa15e3b9c4b6a62cc203439617ac7a2ab6\": container with ID starting with 010f55a43d14481a8b9c383ed944b5fa15e3b9c4b6a62cc203439617ac7a2ab6 not found: ID does not exist"
Oct 29 23:51:40.820186 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.820161    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v" event={"ID":"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32","Type":"ContainerDied","Data":"a958305d467bc69cc3b61cd2dd63e6297fa4d8db5a914e150b3c06c4783e0d9e"}
Oct 29 23:51:40.820307 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.820189    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v" event={"ID":"8c4d02b4-5d2b-466b-a379-fcbdd26b8f32","Type":"ContainerDied","Data":"54044d46db47312e3a64cd758c9f59e519ac324b50b6404d1b1b515f1466b027"}
Oct 29 23:51:40.821432 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.820439    5428 scope.go:122] "RemoveContainer" containerID="a958305d467bc69cc3b61cd2dd63e6297fa4d8db5a914e150b3c06c4783e0d9e"
Oct 29 23:51:40.847059 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.847030    5428 scope.go:122] "RemoveContainer" containerID="a958305d467bc69cc3b61cd2dd63e6297fa4d8db5a914e150b3c06c4783e0d9e"
Oct 29 23:51:40.847625 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:51:40.847600    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a958305d467bc69cc3b61cd2dd63e6297fa4d8db5a914e150b3c06c4783e0d9e\": container with ID starting with a958305d467bc69cc3b61cd2dd63e6297fa4d8db5a914e150b3c06c4783e0d9e not found: ID does not exist" containerID="a958305d467bc69cc3b61cd2dd63e6297fa4d8db5a914e150b3c06c4783e0d9e"
Oct 29 23:51:40.847731 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.847633    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a958305d467bc69cc3b61cd2dd63e6297fa4d8db5a914e150b3c06c4783e0d9e"} err="failed to get container status \"a958305d467bc69cc3b61cd2dd63e6297fa4d8db5a914e150b3c06c4783e0d9e\": rpc error: code = NotFound desc = could not find container \"a958305d467bc69cc3b61cd2dd63e6297fa4d8db5a914e150b3c06c4783e0d9e\": container with ID starting with a958305d467bc69cc3b61cd2dd63e6297fa4d8db5a914e150b3c06c4783e0d9e not found: ID does not exist"
Oct 29 23:51:40.867389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.867359    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b17d6e-4c9f-4067-95b6-124bc1849a60-kube-api-access-bstb6\" (UniqueName: \"kubernetes.io/projected/29b17d6e-4c9f-4067-95b6-124bc1849a60-kube-api-access-bstb6\") pod \"29b17d6e-4c9f-4067-95b6-124bc1849a60\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") "
Oct 29 23:51:40.867550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.867451    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-data\") pod \"29b17d6e-4c9f-4067-95b6-124bc1849a60\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") "
Oct 29 23:51:40.867550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.867486    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-envoy\") pod \"29b17d6e-4c9f-4067-95b6-124bc1849a60\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") "
Oct 29 23:51:40.867550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.867523    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-token\" (UniqueName: \"kubernetes.io/projected/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-token\") pod \"29b17d6e-4c9f-4067-95b6-124bc1849a60\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") "
Oct 29 23:51:40.867736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.867575    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-workload-certs\") pod \"29b17d6e-4c9f-4067-95b6-124bc1849a60\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") "
Oct 29 23:51:40.867736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.867646    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b17d6e-4c9f-4067-95b6-124bc1849a60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b17d6e-4c9f-4067-95b6-124bc1849a60-istiod-ca-cert\") pod \"29b17d6e-4c9f-4067-95b6-124bc1849a60\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") "
Oct 29 23:51:40.867857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.867824    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-data" pod "29b17d6e-4c9f-4067-95b6-124bc1849a60" (UID: "29b17d6e-4c9f-4067-95b6-124bc1849a60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:40.867927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.867832    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-credential-socket\") pod \"29b17d6e-4c9f-4067-95b6-124bc1849a60\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") "
Oct 29 23:51:40.867994 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.867977    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-workload-socket\") pod \"29b17d6e-4c9f-4067-95b6-124bc1849a60\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") "
Oct 29 23:51:40.868065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.868044    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-podinfo\") pod \"29b17d6e-4c9f-4067-95b6-124bc1849a60\" (UID: \"29b17d6e-4c9f-4067-95b6-124bc1849a60\") "
Oct 29 23:51:40.868131 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.868070    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-credential-socket" pod "29b17d6e-4c9f-4067-95b6-124bc1849a60" (UID: "29b17d6e-4c9f-4067-95b6-124bc1849a60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:40.868526 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.868500    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-workload-socket" pod "29b17d6e-4c9f-4067-95b6-124bc1849a60" (UID: "29b17d6e-4c9f-4067-95b6-124bc1849a60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:40.868713 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.868688    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-workload-certs" pod "29b17d6e-4c9f-4067-95b6-124bc1849a60" (UID: "29b17d6e-4c9f-4067-95b6-124bc1849a60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:40.868771 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.868748    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.868771 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.868765    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.868926 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.868775    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.869149 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.869128    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b17d6e-4c9f-4067-95b6-124bc1849a60-istiod-ca-cert" pod "29b17d6e-4c9f-4067-95b6-124bc1849a60" (UID: "29b17d6e-4c9f-4067-95b6-124bc1849a60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:51:40.871364 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.871343    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-envoy" pod "29b17d6e-4c9f-4067-95b6-124bc1849a60" (UID: "29b17d6e-4c9f-4067-95b6-124bc1849a60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:40.871610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.871589    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b17d6e-4c9f-4067-95b6-124bc1849a60-kube-api-access-bstb6" pod "29b17d6e-4c9f-4067-95b6-124bc1849a60" (UID: "29b17d6e-4c9f-4067-95b6-124bc1849a60"). InnerVolumeSpecName "kube-api-access-bstb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:51:40.872931 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.872913    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-podinfo" pod "29b17d6e-4c9f-4067-95b6-124bc1849a60" (UID: "29b17d6e-4c9f-4067-95b6-124bc1849a60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:51:40.873095 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.873072    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-token" pod "29b17d6e-4c9f-4067-95b6-124bc1849a60" (UID: "29b17d6e-4c9f-4067-95b6-124bc1849a60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:51:40.882867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.882822    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:40.896889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.896866    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"]
Oct 29 23:51:40.903848 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.903826    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-klpsg"]
Oct 29 23:51:40.933777 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.933751    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"]
Oct 29 23:51:40.939645 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.939622    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-hbh4v"]
Oct 29 23:51:40.943564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.943534    5428 patch_prober.go:29] interesting pod/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:51:40.943674 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.943582    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr" podUID="97ec11c4-83fa-4ae5-94fa-35a14eed785f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:51:40.969699 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.969674    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.969699 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.969698    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bstb6\" (UniqueName: \"kubernetes.io/projected/29b17d6e-4c9f-4067-95b6-124bc1849a60-kube-api-access-bstb6\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.969887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.969708    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.969887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.969716    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b17d6e-4c9f-4067-95b6-124bc1849a60-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.969887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.969725    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b17d6e-4c9f-4067-95b6-124bc1849a60-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:40.969887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:40.969733    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b17d6e-4c9f-4067-95b6-124bc1849a60-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:41.591622 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:41.591522    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:41.804380 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:41.804349    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"
Oct 29 23:51:41.804580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:41.804389    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz" event={"ID":"29b17d6e-4c9f-4067-95b6-124bc1849a60","Type":"ContainerDied","Data":"1ec8b1f789a96aa6cfd68c8faf69c21adc0b26d13af344714fa111b01b6efc86"}
Oct 29 23:51:41.804580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:41.804441    5428 scope.go:122] "RemoveContainer" containerID="cefba7c9427fa6ac5f4c49431654293361aacdf9bfad42aff77a19ded41d0c8c"
Oct 29 23:51:41.857322 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:41.857263    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"]
Oct 29 23:51:41.869242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:41.869220    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2b0108-openshift-default-89c89b547-8g6gz"]
Oct 29 23:51:42.777587 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:42.777552    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d2f66a5-0f31-4820-9693-b4d87b6c4933" path="/var/lib/kubelet/pods/0d2f66a5-0f31-4820-9693-b4d87b6c4933/volumes"
Oct 29 23:51:42.778154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:42.778139    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b17d6e-4c9f-4067-95b6-124bc1849a60" path="/var/lib/kubelet/pods/29b17d6e-4c9f-4067-95b6-124bc1849a60/volumes"
Oct 29 23:51:42.778638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:42.778625    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" path="/var/lib/kubelet/pods/8c4d02b4-5d2b-466b-a379-fcbdd26b8f32/volumes"
Oct 29 23:51:43.478185 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:43.478156    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:45.196312 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.196276    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:45.858987 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.858949    5428 generic.go:356] "Generic (PLEG): container finished" podID="97ec11c4-83fa-4ae5-94fa-35a14eed785f" containerID="6d28c6b1a570b2d39a78079bc2eb077dbd8fa383f6c04ef7323de76b04d9e893" exitCode=0
Oct 29 23:51:45.859163 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.859139    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr" event={"ID":"97ec11c4-83fa-4ae5-94fa-35a14eed785f","Type":"ContainerDied","Data":"6d28c6b1a570b2d39a78079bc2eb077dbd8fa383f6c04ef7323de76b04d9e893"}
Oct 29 23:51:45.859246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.859232    5428 scope.go:122] "RemoveContainer" containerID="59022cec8fa426e35fe84a9f22ea937326357091f5dcdde4b46ad01a08bc558b"
Oct 29 23:51:45.891376 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.891345    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:51:45.933536 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.933511    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ec11c4-83fa-4ae5-94fa-35a14eed785f-kube-api-access-h5hjh\" (UniqueName: \"kubernetes.io/projected/97ec11c4-83fa-4ae5-94fa-35a14eed785f-kube-api-access-h5hjh\") pod \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") "
Oct 29 23:51:45.933685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.933567    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-workload-socket\") pod \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") "
Oct 29 23:51:45.933685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.933624    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-workload-certs\") pod \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") "
Oct 29 23:51:45.933685 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.933669    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istiod-ca-cert\") pod \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") "
Oct 29 23:51:45.933904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.933696    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-data\") pod \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") "
Oct 29 23:51:45.933904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.933712    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-envoy\") pod \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") "
Oct 29 23:51:45.933904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.933745    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-podinfo\") pod \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") "
Oct 29 23:51:45.933904 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.933788    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-credential-socket\") pod \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") "
Oct 29 23:51:45.934130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.933924    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-token\" (UniqueName: \"kubernetes.io/projected/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-token\") pod \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\" (UID: \"97ec11c4-83fa-4ae5-94fa-35a14eed785f\") "
Oct 29 23:51:45.934130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.933942    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-workload-socket" pod "97ec11c4-83fa-4ae5-94fa-35a14eed785f" (UID: "97ec11c4-83fa-4ae5-94fa-35a14eed785f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:45.934130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.934021    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-workload-certs" pod "97ec11c4-83fa-4ae5-94fa-35a14eed785f" (UID: "97ec11c4-83fa-4ae5-94fa-35a14eed785f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:45.934130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.934040    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-data" pod "97ec11c4-83fa-4ae5-94fa-35a14eed785f" (UID: "97ec11c4-83fa-4ae5-94fa-35a14eed785f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:45.934130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.934119    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istiod-ca-cert" pod "97ec11c4-83fa-4ae5-94fa-35a14eed785f" (UID: "97ec11c4-83fa-4ae5-94fa-35a14eed785f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:51:45.934404 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.934256    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-credential-socket" pod "97ec11c4-83fa-4ae5-94fa-35a14eed785f" (UID: "97ec11c4-83fa-4ae5-94fa-35a14eed785f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:45.934696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.934649    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:45.934696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.934678    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:45.934696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.934696    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:45.934965 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.934709    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:45.934965 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.934722    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:45.935903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.935876    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ec11c4-83fa-4ae5-94fa-35a14eed785f-kube-api-access-h5hjh" pod "97ec11c4-83fa-4ae5-94fa-35a14eed785f" (UID: "97ec11c4-83fa-4ae5-94fa-35a14eed785f"). InnerVolumeSpecName "kube-api-access-h5hjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:51:45.936120 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.936095    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-token" pod "97ec11c4-83fa-4ae5-94fa-35a14eed785f" (UID: "97ec11c4-83fa-4ae5-94fa-35a14eed785f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:51:45.936233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.936210    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-podinfo" pod "97ec11c4-83fa-4ae5-94fa-35a14eed785f" (UID: "97ec11c4-83fa-4ae5-94fa-35a14eed785f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:51:45.936287 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:45.936273    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-envoy" pod "97ec11c4-83fa-4ae5-94fa-35a14eed785f" (UID: "97ec11c4-83fa-4ae5-94fa-35a14eed785f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:46.036104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:46.036060    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5hjh\" (UniqueName: \"kubernetes.io/projected/97ec11c4-83fa-4ae5-94fa-35a14eed785f-kube-api-access-h5hjh\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:46.036104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:46.036096    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:46.036104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:46.036106    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:46.036104 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:46.036115    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ec11c4-83fa-4ae5-94fa-35a14eed785f-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:46.667726 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:46.667695    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:46.792579 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:46.792546    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:46.867004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:46.866979    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"
Oct 29 23:51:46.867171 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:46.867019    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr" event={"ID":"97ec11c4-83fa-4ae5-94fa-35a14eed785f","Type":"ContainerDied","Data":"49fc0c3eff47294242e421c2497eb4e4cd185ce51bb7a87cefc94c073b9ae214"}
Oct 29 23:51:46.867171 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:46.867077    5428 scope.go:122] "RemoveContainer" containerID="6d28c6b1a570b2d39a78079bc2eb077dbd8fa383f6c04ef7323de76b04d9e893"
Oct 29 23:51:46.963859 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:46.963778    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"]
Oct 29 23:51:46.968301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:46.968275    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d20345-openshift-default-54998d6d77-2nkqr"]
Oct 29 23:51:47.371787 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:51:47.371751    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:51:47.383201 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:47.383166    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:47.515695 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:47.515660    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:47.628177 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:47.628062    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:48.003142 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:48.003039    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:48.778286 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:48.778247    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ec11c4-83fa-4ae5-94fa-35a14eed785f" path="/var/lib/kubelet/pods/97ec11c4-83fa-4ae5-94fa-35a14eed785f/volumes"
Oct 29 23:51:49.409724 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:49.409662    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:50.679977 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:50.679944    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g"]
Oct 29 23:51:50.835987 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:50.835955    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:50.954194 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:50.954064    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:51.412625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:51.412590    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:51.544730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:51.544681    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:51.656200 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:51.656167    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:53.315660 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:53.315632    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:54.085640 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:54.085610    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:54.207760 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:54.207724    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:54.417714 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:54.417682    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:54.951773 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:54.951727    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29" podUID="c84aca64-8103-4122-9899-158745522b85" containerName="main" containerID="cri-o://cd62f1ab5c28697edde0eaf3074d837b9ea37123c31f220fefbff796954c0789" gracePeriod=45
Oct 29 23:51:55.229782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.229756    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:51:55.326177 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.326126    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:55.359433 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.359405    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c84aca64-8103-4122-9899-158745522b85-kube-api-access-zcfg9\" (UniqueName: \"kubernetes.io/projected/c84aca64-8103-4122-9899-158745522b85-kube-api-access-zcfg9\") pod \"c84aca64-8103-4122-9899-158745522b85\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") "
Oct 29 23:51:55.359599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.359460    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-dshm\") pod \"c84aca64-8103-4122-9899-158745522b85\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") "
Oct 29 23:51:55.359599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.359545    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-model-cache\") pod \"c84aca64-8103-4122-9899-158745522b85\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") "
Oct 29 23:51:55.359599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.359576    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-home\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-home\") pod \"c84aca64-8103-4122-9899-158745522b85\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") "
Oct 29 23:51:55.359816 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.359642    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c84aca64-8103-4122-9899-158745522b85-tls-certs\" (UniqueName: \"kubernetes.io/secret/c84aca64-8103-4122-9899-158745522b85-tls-certs\") pod \"c84aca64-8103-4122-9899-158745522b85\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") "
Oct 29 23:51:55.359816 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.359765    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-tmp-dir\") pod \"c84aca64-8103-4122-9899-158745522b85\" (UID: \"c84aca64-8103-4122-9899-158745522b85\") "
Oct 29 23:51:55.359946 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.359912    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-model-cache" pod "c84aca64-8103-4122-9899-158745522b85" (UID: "c84aca64-8103-4122-9899-158745522b85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:55.359946 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.359925    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-home" pod "c84aca64-8103-4122-9899-158745522b85" (UID: "c84aca64-8103-4122-9899-158745522b85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:55.360206 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.360184    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-tmp-dir" pod "c84aca64-8103-4122-9899-158745522b85" (UID: "c84aca64-8103-4122-9899-158745522b85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:55.360407 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.360389    5428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-model-cache\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:55.360490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.360410    5428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-home\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:55.360490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.360425    5428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-tmp-dir\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:55.362758 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.362721    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84aca64-8103-4122-9899-158745522b85-kube-api-access-zcfg9" pod "c84aca64-8103-4122-9899-158745522b85" (UID: "c84aca64-8103-4122-9899-158745522b85"). InnerVolumeSpecName "kube-api-access-zcfg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:51:55.362902 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.362852    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84aca64-8103-4122-9899-158745522b85-tls-certs" pod "c84aca64-8103-4122-9899-158745522b85" (UID: "c84aca64-8103-4122-9899-158745522b85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:51:55.364688 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.364666    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-dshm" pod "c84aca64-8103-4122-9899-158745522b85" (UID: "c84aca64-8103-4122-9899-158745522b85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:51:55.462293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.462205    5428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c84aca64-8103-4122-9899-158745522b85-tls-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:55.462293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.462250    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcfg9\" (UniqueName: \"kubernetes.io/projected/c84aca64-8103-4122-9899-158745522b85-kube-api-access-zcfg9\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:55.462293 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.462266    5428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c84aca64-8103-4122-9899-158745522b85-dshm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:51:55.680246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.680208    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g" podUID="b034cafa-0951-4854-899a-a9f03ebb7748" containerName="payload-processing" containerID="cri-o://708786b8043de9be7e43619270ed12cc4b3222f35c13c28ac96a0cb40f5456b9" gracePeriod=25
Oct 29 23:51:55.941954 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.941918    5428 generic.go:356] "Generic (PLEG): container finished" podID="c84aca64-8103-4122-9899-158745522b85" containerID="cd62f1ab5c28697edde0eaf3074d837b9ea37123c31f220fefbff796954c0789" exitCode=0
Oct 29 23:51:55.942132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.941972    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"
Oct 29 23:51:55.942132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.941997    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29" event={"ID":"c84aca64-8103-4122-9899-158745522b85","Type":"ContainerDied","Data":"cd62f1ab5c28697edde0eaf3074d837b9ea37123c31f220fefbff796954c0789"}
Oct 29 23:51:55.942132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.942040    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29" event={"ID":"c84aca64-8103-4122-9899-158745522b85","Type":"ContainerDied","Data":"d2816bbfa89bebd31d3d1d7528484abb5caf34ee998ea0935cb8a12ba70796cb"}
Oct 29 23:51:55.942132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.942057    5428 scope.go:122] "RemoveContainer" containerID="cd62f1ab5c28697edde0eaf3074d837b9ea37123c31f220fefbff796954c0789"
Oct 29 23:51:55.952649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.952465    5428 scope.go:122] "RemoveContainer" containerID="cd62f1ab5c28697edde0eaf3074d837b9ea37123c31f220fefbff796954c0789"
Oct 29 23:51:55.952773 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:51:55.952754    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd62f1ab5c28697edde0eaf3074d837b9ea37123c31f220fefbff796954c0789\": container with ID starting with cd62f1ab5c28697edde0eaf3074d837b9ea37123c31f220fefbff796954c0789 not found: ID does not exist" containerID="cd62f1ab5c28697edde0eaf3074d837b9ea37123c31f220fefbff796954c0789"
Oct 29 23:51:55.952847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.952780    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd62f1ab5c28697edde0eaf3074d837b9ea37123c31f220fefbff796954c0789"} err="failed to get container status \"cd62f1ab5c28697edde0eaf3074d837b9ea37123c31f220fefbff796954c0789\": rpc error: code = NotFound desc = could not find container \"cd62f1ab5c28697edde0eaf3074d837b9ea37123c31f220fefbff796954c0789\": container with ID starting with cd62f1ab5c28697edde0eaf3074d837b9ea37123c31f220fefbff796954c0789 not found: ID does not exist"
Oct 29 23:51:55.994307 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:55.994278    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"]
Oct 29 23:51:56.005261 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:56.005235    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-adc7d7/test-model-adc7d7-kserve-8fdcd9d79-g2t29"]
Oct 29 23:51:56.777174 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:56.777140    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c84aca64-8103-4122-9899-158745522b85" path="/var/lib/kubelet/pods/c84aca64-8103-4122-9899-158745522b85/volumes"
Oct 29 23:51:56.901141 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:56.901108    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:57.678132 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:57.678098    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:57.816265 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:57.816233    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:51:57.933763 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:51:57.933675    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:00.029374 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:00.029342    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:00.875697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:00.875663    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g" podUID="b034cafa-0951-4854-899a-a9f03ebb7748" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:52:01.975177 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:01.975144    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:03.738068 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:03.738014    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:05.344368 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:05.344333    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz"]
Oct 29 23:52:05.489357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:05.489322    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:05.608143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:05.608058    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:07.193860 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:07.193790    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:08.323874 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:08.323836    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:08.984275 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:08.984244    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:10.345198 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:10.345156    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz" podUID="13040ff3-bf4c-4e93-8823-1feb043a8eda" containerName="payload-pre-processing" containerID="cri-o://9e48724303f4691e8d44e8a6e6a2537247c8bf859fab16e749e11bf72b1eaffd" gracePeriod=25
Oct 29 23:52:10.871410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:10.871375    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:10.875287 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:10.875259    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g" podUID="b034cafa-0951-4854-899a-a9f03ebb7748" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:52:12.573468 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:12.573429    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:14.318200 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:14.318166    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:14.660432 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:14.660398    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:14.870023 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:14.869993    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:15.088392 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:15.088312    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:15.970730 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:15.970698    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g"
Oct 29 23:52:16.022721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:16.022690    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b034cafa-0951-4854-899a-a9f03ebb7748-kube-api-access-7dtb8\" (UniqueName: \"kubernetes.io/projected/b034cafa-0951-4854-899a-a9f03ebb7748-kube-api-access-7dtb8\") pod \"b034cafa-0951-4854-899a-a9f03ebb7748\" (UID: \"b034cafa-0951-4854-899a-a9f03ebb7748\") "
Oct 29 23:52:16.022894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:16.022758    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b034cafa-0951-4854-899a-a9f03ebb7748-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b034cafa-0951-4854-899a-a9f03ebb7748-plugins-config-volume\") pod \"b034cafa-0951-4854-899a-a9f03ebb7748\" (UID: \"b034cafa-0951-4854-899a-a9f03ebb7748\") "
Oct 29 23:52:16.023233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:16.023200    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b034cafa-0951-4854-899a-a9f03ebb7748-plugins-config-volume" pod "b034cafa-0951-4854-899a-a9f03ebb7748" (UID: "b034cafa-0951-4854-899a-a9f03ebb7748"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:52:16.023443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:16.023417    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b034cafa-0951-4854-899a-a9f03ebb7748-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:16.025751 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:16.025719    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b034cafa-0951-4854-899a-a9f03ebb7748-kube-api-access-7dtb8" pod "b034cafa-0951-4854-899a-a9f03ebb7748" (UID: "b034cafa-0951-4854-899a-a9f03ebb7748"). InnerVolumeSpecName "kube-api-access-7dtb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:52:16.093459 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:16.093424    5428 generic.go:356] "Generic (PLEG): container finished" podID="b034cafa-0951-4854-899a-a9f03ebb7748" containerID="708786b8043de9be7e43619270ed12cc4b3222f35c13c28ac96a0cb40f5456b9" exitCode=0
Oct 29 23:52:16.093616 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:16.093475    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g"
Oct 29 23:52:16.093616 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:16.093466    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g" event={"ID":"b034cafa-0951-4854-899a-a9f03ebb7748","Type":"ContainerDied","Data":"708786b8043de9be7e43619270ed12cc4b3222f35c13c28ac96a0cb40f5456b9"}
Oct 29 23:52:16.093616 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:16.093599    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g" event={"ID":"b034cafa-0951-4854-899a-a9f03ebb7748","Type":"ContainerDied","Data":"f25d5f2f3d7af7342092f275bdb49ff1a30d217720802fc06c3c49b63973affc"}
Oct 29 23:52:16.093769 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:16.093630    5428 scope.go:122] "RemoveContainer" containerID="708786b8043de9be7e43619270ed12cc4b3222f35c13c28ac96a0cb40f5456b9"
Oct 29 23:52:16.107775 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:16.107750    5428 scope.go:122] "RemoveContainer" containerID="708786b8043de9be7e43619270ed12cc4b3222f35c13c28ac96a0cb40f5456b9"
Oct 29 23:52:16.108108 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:52:16.108075    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"708786b8043de9be7e43619270ed12cc4b3222f35c13c28ac96a0cb40f5456b9\": container with ID starting with 708786b8043de9be7e43619270ed12cc4b3222f35c13c28ac96a0cb40f5456b9 not found: ID does not exist" containerID="708786b8043de9be7e43619270ed12cc4b3222f35c13c28ac96a0cb40f5456b9"
Oct 29 23:52:16.108219 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:16.108119    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"708786b8043de9be7e43619270ed12cc4b3222f35c13c28ac96a0cb40f5456b9"} err="failed to get container status \"708786b8043de9be7e43619270ed12cc4b3222f35c13c28ac96a0cb40f5456b9\": rpc error: code = NotFound desc = could not find container \"708786b8043de9be7e43619270ed12cc4b3222f35c13c28ac96a0cb40f5456b9\": container with ID starting with 708786b8043de9be7e43619270ed12cc4b3222f35c13c28ac96a0cb40f5456b9 not found: ID does not exist"
Oct 29 23:52:16.124451 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:16.124423    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dtb8\" (UniqueName: \"kubernetes.io/projected/b034cafa-0951-4854-899a-a9f03ebb7748-kube-api-access-7dtb8\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:16.145704 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:16.145672    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g"]
Oct 29 23:52:16.149616 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:16.149591    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-37498a-587c8c74fb-dkr7g"]
Oct 29 23:52:16.278241 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:16.278160    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:16.781604 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:16.781571    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b034cafa-0951-4854-899a-a9f03ebb7748" path="/var/lib/kubelet/pods/b034cafa-0951-4854-899a-a9f03ebb7748/volumes"
Oct 29 23:52:16.963185 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:16.963145    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz" podUID="13040ff3-bf4c-4e93-8823-1feb043a8eda" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:52:17.368769 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:52:17.368728    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:52:18.005023 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:18.004991    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:18.121101 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:18.121072    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:18.439694 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:18.439657    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:19.245162 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:19.245126    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:20.064392 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:20.064326    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:20.759696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:20.759663    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:20.877929 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:20.877896    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:20.993159 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:20.993124    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:21.477987 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:21.477952    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:22.900081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:22.900043    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:24.634884 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:24.634844    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:25.130480 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:25.130446    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:25.360676 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:25.360611    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:26.169936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.169907    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:26.193624 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.193593    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"]
Oct 29 23:52:26.193866 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.193797    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt" podUID="b6d6288f-a859-4dee-9fc8-389435a62863" containerName="maas-api" containerID="cri-o://81c6ee577368aadc57f738f40ac1706057157a93780b05ce2eb4bda53ba953aa" gracePeriod=30
Oct 29 23:52:26.315589 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.315561    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:26.475715 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.475684    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"
Oct 29 23:52:26.596466 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.596434    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"]
Oct 29 23:52:26.596693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.596667    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6" podUID="d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec" containerName="maas-api" containerID="cri-o://fb1578059adf619a7ea04c1df22c33ec6d19e4af6985e236c44c652aefda0bab" gracePeriod=30
Oct 29 23:52:26.597496 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.597343    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6d6288f-a859-4dee-9fc8-389435a62863-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d6288f-a859-4dee-9fc8-389435a62863-metrics-certs\") pod \"b6d6288f-a859-4dee-9fc8-389435a62863\" (UID: \"b6d6288f-a859-4dee-9fc8-389435a62863\") "
Oct 29 23:52:26.597580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.597526    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d6288f-a859-4dee-9fc8-389435a62863-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d6288f-a859-4dee-9fc8-389435a62863-openshift-service-ca\") pod \"b6d6288f-a859-4dee-9fc8-389435a62863\" (UID: \"b6d6288f-a859-4dee-9fc8-389435a62863\") "
Oct 29 23:52:26.597648 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.597587    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d6288f-a859-4dee-9fc8-389435a62863-kube-api-access-5tzk5\" (UniqueName: \"kubernetes.io/projected/b6d6288f-a859-4dee-9fc8-389435a62863-kube-api-access-5tzk5\") pod \"b6d6288f-a859-4dee-9fc8-389435a62863\" (UID: \"b6d6288f-a859-4dee-9fc8-389435a62863\") "
Oct 29 23:52:26.597706 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.597678    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6d6288f-a859-4dee-9fc8-389435a62863-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d6288f-a859-4dee-9fc8-389435a62863-maas-api-tls\") pod \"b6d6288f-a859-4dee-9fc8-389435a62863\" (UID: \"b6d6288f-a859-4dee-9fc8-389435a62863\") "
Oct 29 23:52:26.598087 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.598054    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d6288f-a859-4dee-9fc8-389435a62863-openshift-service-ca" pod "b6d6288f-a859-4dee-9fc8-389435a62863" (UID: "b6d6288f-a859-4dee-9fc8-389435a62863"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:52:26.598614 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.598593    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d6288f-a859-4dee-9fc8-389435a62863-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:26.615728 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.614789    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d6288f-a859-4dee-9fc8-389435a62863-maas-api-tls" pod "b6d6288f-a859-4dee-9fc8-389435a62863" (UID: "b6d6288f-a859-4dee-9fc8-389435a62863"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:52:26.615728 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.614899    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d6288f-a859-4dee-9fc8-389435a62863-metrics-certs" pod "b6d6288f-a859-4dee-9fc8-389435a62863" (UID: "b6d6288f-a859-4dee-9fc8-389435a62863"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:52:26.617894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.617856    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d6288f-a859-4dee-9fc8-389435a62863-kube-api-access-5tzk5" pod "b6d6288f-a859-4dee-9fc8-389435a62863" (UID: "b6d6288f-a859-4dee-9fc8-389435a62863"). InnerVolumeSpecName "kube-api-access-5tzk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:52:26.679212 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.679168    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:26.700351 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.700280    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d6288f-a859-4dee-9fc8-389435a62863-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:26.700351 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.700308    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d6288f-a859-4dee-9fc8-389435a62863-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:26.700351 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.700318    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tzk5\" (UniqueName: \"kubernetes.io/projected/b6d6288f-a859-4dee-9fc8-389435a62863-kube-api-access-5tzk5\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:26.795992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.795959    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:26.877164 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.877138    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"
Oct 29 23:52:26.907837 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.907784    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:26.964333 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:26.964251    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz" podUID="13040ff3-bf4c-4e93-8823-1feb043a8eda" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:52:27.003992 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.003962    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-metrics-certs\") pod \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\" (UID: \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\") "
Oct 29 23:52:27.004169 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.004025    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-kube-api-access-555kf\" (UniqueName: \"kubernetes.io/projected/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-kube-api-access-555kf\") pod \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\" (UID: \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\") "
Oct 29 23:52:27.004169 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.004061    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-maas-api-tls\") pod \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\" (UID: \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\") "
Oct 29 23:52:27.004311 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.004258    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-openshift-service-ca\") pod \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\" (UID: \"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec\") "
Oct 29 23:52:27.004633 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.004606    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-openshift-service-ca" pod "d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec" (UID: "d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:52:27.004910 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.004891    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:27.006158 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.006139    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-metrics-certs" pod "d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec" (UID: "d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:52:27.006615 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.006588    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-maas-api-tls" pod "d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec" (UID: "d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:52:27.006615 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.006597    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-kube-api-access-555kf" pod "d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec" (UID: "d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec"). InnerVolumeSpecName "kube-api-access-555kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:52:27.112011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.111928    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:27.112011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.111964    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-555kf\" (UniqueName: \"kubernetes.io/projected/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-kube-api-access-555kf\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:27.112011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.111980    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:27.186109 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.186068    5428 generic.go:356] "Generic (PLEG): container finished" podID="b6d6288f-a859-4dee-9fc8-389435a62863" containerID="81c6ee577368aadc57f738f40ac1706057157a93780b05ce2eb4bda53ba953aa" exitCode=0
Oct 29 23:52:27.186109 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.186110    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"
Oct 29 23:52:27.186673 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.186153    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt" event={"ID":"b6d6288f-a859-4dee-9fc8-389435a62863","Type":"ContainerDied","Data":"81c6ee577368aadc57f738f40ac1706057157a93780b05ce2eb4bda53ba953aa"}
Oct 29 23:52:27.186673 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.186212    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt" event={"ID":"b6d6288f-a859-4dee-9fc8-389435a62863","Type":"ContainerDied","Data":"dcf8c9a0c9f06fee439ade6b89806e8432d8201dbd292ed9d0dcbe7fc7398afb"}
Oct 29 23:52:27.186673 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.186249    5428 scope.go:122] "RemoveContainer" containerID="81c6ee577368aadc57f738f40ac1706057157a93780b05ce2eb4bda53ba953aa"
Oct 29 23:52:27.188188 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.188144    5428 generic.go:356] "Generic (PLEG): container finished" podID="d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec" containerID="fb1578059adf619a7ea04c1df22c33ec6d19e4af6985e236c44c652aefda0bab" exitCode=0
Oct 29 23:52:27.188317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.188181    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6" event={"ID":"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec","Type":"ContainerDied","Data":"fb1578059adf619a7ea04c1df22c33ec6d19e4af6985e236c44c652aefda0bab"}
Oct 29 23:52:27.188317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.188200    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"
Oct 29 23:52:27.188317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.188235    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6" event={"ID":"d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec","Type":"ContainerDied","Data":"32403120b0295585caff37571b08df70867c4ac771854e5437c614c5baf7f1cb"}
Oct 29 23:52:27.233721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.233700    5428 scope.go:122] "RemoveContainer" containerID="81c6ee577368aadc57f738f40ac1706057157a93780b05ce2eb4bda53ba953aa"
Oct 29 23:52:27.234090 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:52:27.234068    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c6ee577368aadc57f738f40ac1706057157a93780b05ce2eb4bda53ba953aa\": container with ID starting with 81c6ee577368aadc57f738f40ac1706057157a93780b05ce2eb4bda53ba953aa not found: ID does not exist" containerID="81c6ee577368aadc57f738f40ac1706057157a93780b05ce2eb4bda53ba953aa"
Oct 29 23:52:27.234153 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.234099    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c6ee577368aadc57f738f40ac1706057157a93780b05ce2eb4bda53ba953aa"} err="failed to get container status \"81c6ee577368aadc57f738f40ac1706057157a93780b05ce2eb4bda53ba953aa\": rpc error: code = NotFound desc = could not find container \"81c6ee577368aadc57f738f40ac1706057157a93780b05ce2eb4bda53ba953aa\": container with ID starting with 81c6ee577368aadc57f738f40ac1706057157a93780b05ce2eb4bda53ba953aa not found: ID does not exist"
Oct 29 23:52:27.234153 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.234116    5428 scope.go:122] "RemoveContainer" containerID="fb1578059adf619a7ea04c1df22c33ec6d19e4af6985e236c44c652aefda0bab"
Oct 29 23:52:27.244010 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.243986    5428 scope.go:122] "RemoveContainer" containerID="fb1578059adf619a7ea04c1df22c33ec6d19e4af6985e236c44c652aefda0bab"
Oct 29 23:52:27.244267 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:52:27.244247    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1578059adf619a7ea04c1df22c33ec6d19e4af6985e236c44c652aefda0bab\": container with ID starting with fb1578059adf619a7ea04c1df22c33ec6d19e4af6985e236c44c652aefda0bab not found: ID does not exist" containerID="fb1578059adf619a7ea04c1df22c33ec6d19e4af6985e236c44c652aefda0bab"
Oct 29 23:52:27.244331 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.244277    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1578059adf619a7ea04c1df22c33ec6d19e4af6985e236c44c652aefda0bab"} err="failed to get container status \"fb1578059adf619a7ea04c1df22c33ec6d19e4af6985e236c44c652aefda0bab\": rpc error: code = NotFound desc = could not find container \"fb1578059adf619a7ea04c1df22c33ec6d19e4af6985e236c44c652aefda0bab\": container with ID starting with fb1578059adf619a7ea04c1df22c33ec6d19e4af6985e236c44c652aefda0bab not found: ID does not exist"
Oct 29 23:52:27.262740 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.262709    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"]
Oct 29 23:52:27.262909 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.262764    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-825472-5b95789775-q6stt"]
Oct 29 23:52:27.292919 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.292871    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"]
Oct 29 23:52:27.297985 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.297955    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-206095-5bfb5c94c5-xm6c6"]
Oct 29 23:52:27.399210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.399174    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:27.507054 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:27.506971    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:28.229278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:28.229248    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:28.360181 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:28.360139    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:28.564197 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:28.564117    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:28.777062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:28.777027    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d6288f-a859-4dee-9fc8-389435a62863" path="/var/lib/kubelet/pods/b6d6288f-a859-4dee-9fc8-389435a62863/volumes"
Oct 29 23:52:28.777493 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:28.777479    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec" path="/var/lib/kubelet/pods/d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec/volumes"
Oct 29 23:52:29.603106 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:29.603073    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:30.648612 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:30.648578    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz"
Oct 29 23:52:30.757332 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:30.757299    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13040ff3-bf4c-4e93-8823-1feb043a8eda-kube-api-access-l8xc5\" (UniqueName: \"kubernetes.io/projected/13040ff3-bf4c-4e93-8823-1feb043a8eda-kube-api-access-l8xc5\") pod \"13040ff3-bf4c-4e93-8823-1feb043a8eda\" (UID: \"13040ff3-bf4c-4e93-8823-1feb043a8eda\") "
Oct 29 23:52:30.757520 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:30.757448    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13040ff3-bf4c-4e93-8823-1feb043a8eda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13040ff3-bf4c-4e93-8823-1feb043a8eda-plugins-config-volume\") pod \"13040ff3-bf4c-4e93-8823-1feb043a8eda\" (UID: \"13040ff3-bf4c-4e93-8823-1feb043a8eda\") "
Oct 29 23:52:30.757856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:30.757828    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13040ff3-bf4c-4e93-8823-1feb043a8eda-plugins-config-volume" pod "13040ff3-bf4c-4e93-8823-1feb043a8eda" (UID: "13040ff3-bf4c-4e93-8823-1feb043a8eda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:52:30.759670 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:30.759643    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13040ff3-bf4c-4e93-8823-1feb043a8eda-kube-api-access-l8xc5" pod "13040ff3-bf4c-4e93-8823-1feb043a8eda" (UID: "13040ff3-bf4c-4e93-8823-1feb043a8eda"). InnerVolumeSpecName "kube-api-access-l8xc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:52:30.858592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:30.858569    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8xc5\" (UniqueName: \"kubernetes.io/projected/13040ff3-bf4c-4e93-8823-1feb043a8eda-kube-api-access-l8xc5\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:30.858592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:30.858591    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13040ff3-bf4c-4e93-8823-1feb043a8eda-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:31.219128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:31.219088    5428 generic.go:356] "Generic (PLEG): container finished" podID="13040ff3-bf4c-4e93-8823-1feb043a8eda" containerID="9e48724303f4691e8d44e8a6e6a2537247c8bf859fab16e749e11bf72b1eaffd" exitCode=0
Oct 29 23:52:31.219342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:31.219143    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz"
Oct 29 23:52:31.219342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:31.219168    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz" event={"ID":"13040ff3-bf4c-4e93-8823-1feb043a8eda","Type":"ContainerDied","Data":"9e48724303f4691e8d44e8a6e6a2537247c8bf859fab16e749e11bf72b1eaffd"}
Oct 29 23:52:31.219342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:31.219224    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz" event={"ID":"13040ff3-bf4c-4e93-8823-1feb043a8eda","Type":"ContainerDied","Data":"25aef15166d097474154c05c7d189bec9daa668444c1e84836cba516fc939567"}
Oct 29 23:52:31.219342 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:31.219248    5428 scope.go:122] "RemoveContainer" containerID="9e48724303f4691e8d44e8a6e6a2537247c8bf859fab16e749e11bf72b1eaffd"
Oct 29 23:52:31.230325 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:31.230307    5428 scope.go:122] "RemoveContainer" containerID="9e48724303f4691e8d44e8a6e6a2537247c8bf859fab16e749e11bf72b1eaffd"
Oct 29 23:52:31.230624 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:52:31.230599    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e48724303f4691e8d44e8a6e6a2537247c8bf859fab16e749e11bf72b1eaffd\": container with ID starting with 9e48724303f4691e8d44e8a6e6a2537247c8bf859fab16e749e11bf72b1eaffd not found: ID does not exist" containerID="9e48724303f4691e8d44e8a6e6a2537247c8bf859fab16e749e11bf72b1eaffd"
Oct 29 23:52:31.230695 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:31.230633    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e48724303f4691e8d44e8a6e6a2537247c8bf859fab16e749e11bf72b1eaffd"} err="failed to get container status \"9e48724303f4691e8d44e8a6e6a2537247c8bf859fab16e749e11bf72b1eaffd\": rpc error: code = NotFound desc = could not find container \"9e48724303f4691e8d44e8a6e6a2537247c8bf859fab16e749e11bf72b1eaffd\": container with ID starting with 9e48724303f4691e8d44e8a6e6a2537247c8bf859fab16e749e11bf72b1eaffd not found: ID does not exist"
Oct 29 23:52:31.280373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:31.280346    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz"]
Oct 29 23:52:31.283242 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:31.283219    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-825472-55fd6dc64c-lv8qz"]
Oct 29 23:52:31.484765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:31.484667    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:32.776789 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:32.776753    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13040ff3-bf4c-4e93-8823-1feb043a8eda" path="/var/lib/kubelet/pods/13040ff3-bf4c-4e93-8823-1feb043a8eda/volumes"
Oct 29 23:52:33.289295 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:33.289262    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:33.421849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:33.419643    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:34.019487 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:34.019451    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:34.955849 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:34.955810    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:35.086468 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:35.086422    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:36.965160 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:36.965122    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:38.681004 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:38.680968    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:39.664270 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:39.664233    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs"]
Oct 29 23:52:39.695332 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:39.695299    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh"]
Oct 29 23:52:39.803431 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:39.803396    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:39.917061 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:39.916981    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:40.503547 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:40.503510    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:42.098863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:42.098829    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:42.635864 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:42.635824    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:42.765133 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:42.765099    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:42.878366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:42.878331    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:43.294937 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:43.294908    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:43.629964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:43.629934    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"]
Oct 29 23:52:43.630152 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:43.630128    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr" podUID="dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" containerName="istio-proxy" containerID="cri-o://11f1d4be6b1f7bf59285dce0c10d3431ba5846455094582097409f5923cffbfd" gracePeriod=30
Oct 29 23:52:43.641612 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:43.641587    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"]
Oct 29 23:52:43.641784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:43.641766    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2" podUID="772a13d1-e842-4cf0-b36e-09e07d7936e4" containerName="istio-proxy" containerID="cri-o://dc89684e7e7619f1c137f31d6f861fc3db539886352a9ee2f2f431f0522cb77d" gracePeriod=30
Oct 29 23:52:43.653703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:43.653676    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"]
Oct 29 23:52:43.653890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:43.653868    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h" podUID="43dcb139-963e-4266-852b-d0119b2d7b0e" containerName="istio-proxy" containerID="cri-o://db1e30c2e05fbf80c359e6d1e6037e6a0345f7450dff31bf8c79bcb3f424d66b" gracePeriod=30
Oct 29 23:52:43.753491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:43.753457    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:44.664618 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:44.664583    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs" podUID="85e2d99a-031b-441f-b881-df20fba27bba" containerName="payload-processing" containerID="cri-o://b2e41f5010f5de32a01a592a9b4140ad68797292d418202e8361d6d457055837" gracePeriod=25
Oct 29 23:52:44.696490 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:44.696461    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh" podUID="e5d61a10-3f0c-42b8-a289-3c0830622387" containerName="payload-pre-processing" containerID="cri-o://ec1eb6dabcd3d09d128fde48fea76ac9a9430911954c5ecbc3a7f921aade5d4f" gracePeriod=25
Oct 29 23:52:44.774400 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:44.774366    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:44.932161 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:44.932092    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs"
Oct 29 23:52:45.036307 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:45.036273    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e2d99a-031b-441f-b881-df20fba27bba-kube-api-access-8lcnd\" (UniqueName: \"kubernetes.io/projected/85e2d99a-031b-441f-b881-df20fba27bba-kube-api-access-8lcnd\") pod \"85e2d99a-031b-441f-b881-df20fba27bba\" (UID: \"85e2d99a-031b-441f-b881-df20fba27bba\") "
Oct 29 23:52:45.036504 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:45.036319    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85e2d99a-031b-441f-b881-df20fba27bba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85e2d99a-031b-441f-b881-df20fba27bba-plugins-config-volume\") pod \"85e2d99a-031b-441f-b881-df20fba27bba\" (UID: \"85e2d99a-031b-441f-b881-df20fba27bba\") "
Oct 29 23:52:45.036756 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:45.036731    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e2d99a-031b-441f-b881-df20fba27bba-plugins-config-volume" pod "85e2d99a-031b-441f-b881-df20fba27bba" (UID: "85e2d99a-031b-441f-b881-df20fba27bba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:52:45.036998 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:45.036979    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85e2d99a-031b-441f-b881-df20fba27bba-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:45.038519 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:45.038502    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e2d99a-031b-441f-b881-df20fba27bba-kube-api-access-8lcnd" pod "85e2d99a-031b-441f-b881-df20fba27bba" (UID: "85e2d99a-031b-441f-b881-df20fba27bba"). InnerVolumeSpecName "kube-api-access-8lcnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:52:45.137965 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:45.137927    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lcnd\" (UniqueName: \"kubernetes.io/projected/85e2d99a-031b-441f-b881-df20fba27bba-kube-api-access-8lcnd\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:45.326165 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:45.326075    5428 generic.go:356] "Generic (PLEG): container finished" podID="85e2d99a-031b-441f-b881-df20fba27bba" containerID="b2e41f5010f5de32a01a592a9b4140ad68797292d418202e8361d6d457055837" exitCode=0
Oct 29 23:52:45.326165 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:45.326127    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs"
Oct 29 23:52:45.326165 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:45.326157    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs" event={"ID":"85e2d99a-031b-441f-b881-df20fba27bba","Type":"ContainerDied","Data":"b2e41f5010f5de32a01a592a9b4140ad68797292d418202e8361d6d457055837"}
Oct 29 23:52:45.326467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:45.326208    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs" event={"ID":"85e2d99a-031b-441f-b881-df20fba27bba","Type":"ContainerDied","Data":"b5979681ecf2ced5be19188d58c75f54215072f3cc73b10c77b781192b68b3cc"}
Oct 29 23:52:45.326467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:45.326230    5428 scope.go:122] "RemoveContainer" containerID="b2e41f5010f5de32a01a592a9b4140ad68797292d418202e8361d6d457055837"
Oct 29 23:52:45.345176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:45.345110    5428 scope.go:122] "RemoveContainer" containerID="b2e41f5010f5de32a01a592a9b4140ad68797292d418202e8361d6d457055837"
Oct 29 23:52:45.345711 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:52:45.345682    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e41f5010f5de32a01a592a9b4140ad68797292d418202e8361d6d457055837\": container with ID starting with b2e41f5010f5de32a01a592a9b4140ad68797292d418202e8361d6d457055837 not found: ID does not exist" containerID="b2e41f5010f5de32a01a592a9b4140ad68797292d418202e8361d6d457055837"
Oct 29 23:52:45.345813 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:45.345718    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e41f5010f5de32a01a592a9b4140ad68797292d418202e8361d6d457055837"} err="failed to get container status \"b2e41f5010f5de32a01a592a9b4140ad68797292d418202e8361d6d457055837\": rpc error: code = NotFound desc = could not find container \"b2e41f5010f5de32a01a592a9b4140ad68797292d418202e8361d6d457055837\": container with ID starting with b2e41f5010f5de32a01a592a9b4140ad68797292d418202e8361d6d457055837 not found: ID does not exist"
Oct 29 23:52:45.379670 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:45.379645    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs"]
Oct 29 23:52:45.382485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:45.382465    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-adc7d7-67dfc5b99d-bdgbs"]
Oct 29 23:52:46.359459 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:46.359414    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:46.777584 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:46.777548    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85e2d99a-031b-441f-b881-df20fba27bba" path="/var/lib/kubelet/pods/85e2d99a-031b-441f-b881-df20fba27bba/volumes"
Oct 29 23:52:47.010564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:47.010530    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:47.141712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:47.141681    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:47.367450 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:52:47.367409    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:52:47.721775 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:47.721739    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:48.973698 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:48.973679    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:52:48.996187 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:48.996156    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:52:49.000475 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.000446    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:52:49.088647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.088412    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/772a13d1-e842-4cf0-b36e-09e07d7936e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772a13d1-e842-4cf0-b36e-09e07d7936e4-istiod-ca-cert\") pod \"772a13d1-e842-4cf0-b36e-09e07d7936e4\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") "
Oct 29 23:52:49.088647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.088464    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772a13d1-e842-4cf0-b36e-09e07d7936e4-kube-api-access-qf9lw\" (UniqueName: \"kubernetes.io/projected/772a13d1-e842-4cf0-b36e-09e07d7936e4-kube-api-access-qf9lw\") pod \"772a13d1-e842-4cf0-b36e-09e07d7936e4\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") "
Oct 29 23:52:49.088647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.088505    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-podinfo\") pod \"43dcb139-963e-4266-852b-d0119b2d7b0e\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") "
Oct 29 23:52:49.088647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.088535    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-envoy\") pod \"43dcb139-963e-4266-852b-d0119b2d7b0e\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") "
Oct 29 23:52:49.088647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.088564    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-credential-socket\") pod \"43dcb139-963e-4266-852b-d0119b2d7b0e\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") "
Oct 29 23:52:49.088647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.088596    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-kube-api-access-p74q7\" (UniqueName: \"kubernetes.io/projected/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-kube-api-access-p74q7\") pod \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") "
Oct 29 23:52:49.090208 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.090175    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772a13d1-e842-4cf0-b36e-09e07d7936e4-istiod-ca-cert" pod "772a13d1-e842-4cf0-b36e-09e07d7936e4" (UID: "772a13d1-e842-4cf0-b36e-09e07d7936e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:52:49.090505 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.090481    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-credential-socket" pod "43dcb139-963e-4266-852b-d0119b2d7b0e" (UID: "43dcb139-963e-4266-852b-d0119b2d7b0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:49.090591 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.090564    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istiod-ca-cert\") pod \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") "
Oct 29 23:52:49.090658 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.090615    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-workload-socket\") pod \"43dcb139-963e-4266-852b-d0119b2d7b0e\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") "
Oct 29 23:52:49.090723 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.090657    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-podinfo\") pod \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") "
Oct 29 23:52:49.090723 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.090685    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-workload-certs\") pod \"43dcb139-963e-4266-852b-d0119b2d7b0e\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") "
Oct 29 23:52:49.090889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.090721    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-workload-socket\") pod \"772a13d1-e842-4cf0-b36e-09e07d7936e4\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") "
Oct 29 23:52:49.090889 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.090794    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-token\" (UniqueName: \"kubernetes.io/projected/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-token\") pod \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") "
Oct 29 23:52:49.091017 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.090896    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-workload-socket\") pod \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") "
Oct 29 23:52:49.091017 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.090921    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-podinfo\") pod \"772a13d1-e842-4cf0-b36e-09e07d7936e4\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") "
Oct 29 23:52:49.091017 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.090949    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-data\") pod \"772a13d1-e842-4cf0-b36e-09e07d7936e4\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") "
Oct 29 23:52:49.091017 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.090985    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-credential-socket\") pod \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") "
Oct 29 23:52:49.091017 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.091008    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-envoy\") pod \"772a13d1-e842-4cf0-b36e-09e07d7936e4\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") "
Oct 29 23:52:49.091280 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.091047    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-workload-certs\") pod \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") "
Oct 29 23:52:49.091280 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.091066    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-data\") pod \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") "
Oct 29 23:52:49.091280 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.091130    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-token\" (UniqueName: \"kubernetes.io/projected/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-token\") pod \"772a13d1-e842-4cf0-b36e-09e07d7936e4\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") "
Oct 29 23:52:49.091280 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.091157    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43dcb139-963e-4266-852b-d0119b2d7b0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43dcb139-963e-4266-852b-d0119b2d7b0e-istiod-ca-cert\") pod \"43dcb139-963e-4266-852b-d0119b2d7b0e\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") "
Oct 29 23:52:49.091280 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.091209    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-data\") pod \"43dcb139-963e-4266-852b-d0119b2d7b0e\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") "
Oct 29 23:52:49.091280 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.091237    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43dcb139-963e-4266-852b-d0119b2d7b0e-kube-api-access-bp48x\" (UniqueName: \"kubernetes.io/projected/43dcb139-963e-4266-852b-d0119b2d7b0e-kube-api-access-bp48x\") pod \"43dcb139-963e-4266-852b-d0119b2d7b0e\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") "
Oct 29 23:52:49.091280 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.091264    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-token\" (UniqueName: \"kubernetes.io/projected/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-token\") pod \"43dcb139-963e-4266-852b-d0119b2d7b0e\" (UID: \"43dcb139-963e-4266-852b-d0119b2d7b0e\") "
Oct 29 23:52:49.091579 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.091295    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-envoy\") pod \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\" (UID: \"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c\") "
Oct 29 23:52:49.091579 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.091323    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-workload-certs\") pod \"772a13d1-e842-4cf0-b36e-09e07d7936e4\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") "
Oct 29 23:52:49.091579 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.091342    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-credential-socket\") pod \"772a13d1-e842-4cf0-b36e-09e07d7936e4\" (UID: \"772a13d1-e842-4cf0-b36e-09e07d7936e4\") "
Oct 29 23:52:49.092096 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.092067    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-credential-socket" pod "dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" (UID: "dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:49.092393 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.092187    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-workload-socket" pod "dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" (UID: "dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:49.092932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.092902    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-workload-socket" pod "772a13d1-e842-4cf0-b36e-09e07d7936e4" (UID: "772a13d1-e842-4cf0-b36e-09e07d7936e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:49.093610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.093584    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-kube-api-access-p74q7" pod "dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" (UID: "dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c"). InnerVolumeSpecName "kube-api-access-p74q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:52:49.094509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.094461    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istiod-ca-cert" pod "dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" (UID: "dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:52:49.094784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.094759    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-workload-certs" pod "43dcb139-963e-4266-852b-d0119b2d7b0e" (UID: "43dcb139-963e-4266-852b-d0119b2d7b0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:49.095659 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.095634    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-workload-certs" pod "dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" (UID: "dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:49.095970 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.095951    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-data" pod "dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" (UID: "dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:49.096109 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.096074    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-data" pod "772a13d1-e842-4cf0-b36e-09e07d7936e4" (UID: "772a13d1-e842-4cf0-b36e-09e07d7936e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:49.096646 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.096569    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:49.096971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.096947    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-envoy" pod "772a13d1-e842-4cf0-b36e-09e07d7936e4" (UID: "772a13d1-e842-4cf0-b36e-09e07d7936e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:49.097171 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.097138    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-workload-socket" pod "43dcb139-963e-4266-852b-d0119b2d7b0e" (UID: "43dcb139-963e-4266-852b-d0119b2d7b0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:49.097461 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.097439    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/772a13d1-e842-4cf0-b36e-09e07d7936e4-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.097579 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.097470    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.098165 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.098138    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-data" pod "43dcb139-963e-4266-852b-d0119b2d7b0e" (UID: "43dcb139-963e-4266-852b-d0119b2d7b0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:49.098268 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.098217    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-workload-certs" pod "772a13d1-e842-4cf0-b36e-09e07d7936e4" (UID: "772a13d1-e842-4cf0-b36e-09e07d7936e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:49.098404 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.098383    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43dcb139-963e-4266-852b-d0119b2d7b0e-istiod-ca-cert" pod "43dcb139-963e-4266-852b-d0119b2d7b0e" (UID: "43dcb139-963e-4266-852b-d0119b2d7b0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:52:49.098474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.098451    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-credential-socket" pod "772a13d1-e842-4cf0-b36e-09e07d7936e4" (UID: "772a13d1-e842-4cf0-b36e-09e07d7936e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:49.099052 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.098996    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-podinfo" pod "772a13d1-e842-4cf0-b36e-09e07d7936e4" (UID: "772a13d1-e842-4cf0-b36e-09e07d7936e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:52:49.100635 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.100593    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-token" pod "dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" (UID: "dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:52:49.101272 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.101249    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-envoy" pod "dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" (UID: "dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:49.101386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.101253    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-token" pod "43dcb139-963e-4266-852b-d0119b2d7b0e" (UID: "43dcb139-963e-4266-852b-d0119b2d7b0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:52:49.101693 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.101659    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-token" pod "772a13d1-e842-4cf0-b36e-09e07d7936e4" (UID: "772a13d1-e842-4cf0-b36e-09e07d7936e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:52:49.102425 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.102400    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-podinfo" pod "dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" (UID: "dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:52:49.103043 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.103019    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43dcb139-963e-4266-852b-d0119b2d7b0e-kube-api-access-bp48x" pod "43dcb139-963e-4266-852b-d0119b2d7b0e" (UID: "43dcb139-963e-4266-852b-d0119b2d7b0e"). InnerVolumeSpecName "kube-api-access-bp48x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:52:49.103157 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.103138    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-envoy" pod "43dcb139-963e-4266-852b-d0119b2d7b0e" (UID: "43dcb139-963e-4266-852b-d0119b2d7b0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:49.103326 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.103302    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-podinfo" pod "43dcb139-963e-4266-852b-d0119b2d7b0e" (UID: "43dcb139-963e-4266-852b-d0119b2d7b0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:52:49.104736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.104714    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772a13d1-e842-4cf0-b36e-09e07d7936e4-kube-api-access-qf9lw" pod "772a13d1-e842-4cf0-b36e-09e07d7936e4" (UID: "772a13d1-e842-4cf0-b36e-09e07d7936e4"). InnerVolumeSpecName "kube-api-access-qf9lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:52:49.198438 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198348    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198438 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198376    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198438 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198386    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198438 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198394    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43dcb139-963e-4266-852b-d0119b2d7b0e-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198438 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198403    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198438 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198412    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp48x\" (UniqueName: \"kubernetes.io/projected/43dcb139-963e-4266-852b-d0119b2d7b0e-kube-api-access-bp48x\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198438 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198421    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198438 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198429    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198438 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198437    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198438 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198444    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198452    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf9lw\" (UniqueName: \"kubernetes.io/projected/772a13d1-e842-4cf0-b36e-09e07d7936e4-kube-api-access-qf9lw\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198460    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198468    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198477    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p74q7\" (UniqueName: \"kubernetes.io/projected/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-kube-api-access-p74q7\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198485    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198493    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198501    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198508    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43dcb139-963e-4266-852b-d0119b2d7b0e-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198516    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198524    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198532    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198540    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198548    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198556    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.198932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.198563    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/772a13d1-e842-4cf0-b36e-09e07d7936e4-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:49.357767 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.357735    5428 generic.go:356] "Generic (PLEG): container finished" podID="43dcb139-963e-4266-852b-d0119b2d7b0e" containerID="db1e30c2e05fbf80c359e6d1e6037e6a0345f7450dff31bf8c79bcb3f424d66b" exitCode=0
Oct 29 23:52:49.357982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.357785    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"
Oct 29 23:52:49.357982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.357825    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h" event={"ID":"43dcb139-963e-4266-852b-d0119b2d7b0e","Type":"ContainerDied","Data":"db1e30c2e05fbf80c359e6d1e6037e6a0345f7450dff31bf8c79bcb3f424d66b"}
Oct 29 23:52:49.357982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.357885    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h" event={"ID":"43dcb139-963e-4266-852b-d0119b2d7b0e","Type":"ContainerDied","Data":"c19121fcc8df3e92bfcbee5a9a18a9c807a8db3fab0126738c0b821f1f68f5cb"}
Oct 29 23:52:49.357982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.357907    5428 scope.go:122] "RemoveContainer" containerID="db1e30c2e05fbf80c359e6d1e6037e6a0345f7450dff31bf8c79bcb3f424d66b"
Oct 29 23:52:49.359473 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.359445    5428 generic.go:356] "Generic (PLEG): container finished" podID="dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" containerID="11f1d4be6b1f7bf59285dce0c10d3431ba5846455094582097409f5923cffbfd" exitCode=0
Oct 29 23:52:49.359625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.359494    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"
Oct 29 23:52:49.359625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.359547    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr" event={"ID":"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c","Type":"ContainerDied","Data":"11f1d4be6b1f7bf59285dce0c10d3431ba5846455094582097409f5923cffbfd"}
Oct 29 23:52:49.359625 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.359599    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr" event={"ID":"dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c","Type":"ContainerDied","Data":"4da447c210cbb214f8b7fdc257c303c832fa839fbc6e3d2364042722645794a4"}
Oct 29 23:52:49.361845 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.361789    5428 generic.go:356] "Generic (PLEG): container finished" podID="772a13d1-e842-4cf0-b36e-09e07d7936e4" containerID="dc89684e7e7619f1c137f31d6f861fc3db539886352a9ee2f2f431f0522cb77d" exitCode=0
Oct 29 23:52:49.361982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.361860    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2" event={"ID":"772a13d1-e842-4cf0-b36e-09e07d7936e4","Type":"ContainerDied","Data":"dc89684e7e7619f1c137f31d6f861fc3db539886352a9ee2f2f431f0522cb77d"}
Oct 29 23:52:49.361982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.361893    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2" event={"ID":"772a13d1-e842-4cf0-b36e-09e07d7936e4","Type":"ContainerDied","Data":"070f1655c4cec3cba1e1e3b2006fe1cd0774113dd9bb4e9fdf76710172bbbbcd"}
Oct 29 23:52:49.361982 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.361946    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"
Oct 29 23:52:49.371592 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.371559    5428 scope.go:122] "RemoveContainer" containerID="db1e30c2e05fbf80c359e6d1e6037e6a0345f7450dff31bf8c79bcb3f424d66b"
Oct 29 23:52:49.379079 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:52:49.376131    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db1e30c2e05fbf80c359e6d1e6037e6a0345f7450dff31bf8c79bcb3f424d66b\": container with ID starting with db1e30c2e05fbf80c359e6d1e6037e6a0345f7450dff31bf8c79bcb3f424d66b not found: ID does not exist" containerID="db1e30c2e05fbf80c359e6d1e6037e6a0345f7450dff31bf8c79bcb3f424d66b"
Oct 29 23:52:49.379079 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.376171    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db1e30c2e05fbf80c359e6d1e6037e6a0345f7450dff31bf8c79bcb3f424d66b"} err="failed to get container status \"db1e30c2e05fbf80c359e6d1e6037e6a0345f7450dff31bf8c79bcb3f424d66b\": rpc error: code = NotFound desc = could not find container \"db1e30c2e05fbf80c359e6d1e6037e6a0345f7450dff31bf8c79bcb3f424d66b\": container with ID starting with db1e30c2e05fbf80c359e6d1e6037e6a0345f7450dff31bf8c79bcb3f424d66b not found: ID does not exist"
Oct 29 23:52:49.379079 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.376196    5428 scope.go:122] "RemoveContainer" containerID="11f1d4be6b1f7bf59285dce0c10d3431ba5846455094582097409f5923cffbfd"
Oct 29 23:52:49.392921 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.392902    5428 scope.go:122] "RemoveContainer" containerID="11f1d4be6b1f7bf59285dce0c10d3431ba5846455094582097409f5923cffbfd"
Oct 29 23:52:49.393197 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:52:49.393169    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f1d4be6b1f7bf59285dce0c10d3431ba5846455094582097409f5923cffbfd\": container with ID starting with 11f1d4be6b1f7bf59285dce0c10d3431ba5846455094582097409f5923cffbfd not found: ID does not exist" containerID="11f1d4be6b1f7bf59285dce0c10d3431ba5846455094582097409f5923cffbfd"
Oct 29 23:52:49.393269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.393210    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f1d4be6b1f7bf59285dce0c10d3431ba5846455094582097409f5923cffbfd"} err="failed to get container status \"11f1d4be6b1f7bf59285dce0c10d3431ba5846455094582097409f5923cffbfd\": rpc error: code = NotFound desc = could not find container \"11f1d4be6b1f7bf59285dce0c10d3431ba5846455094582097409f5923cffbfd\": container with ID starting with 11f1d4be6b1f7bf59285dce0c10d3431ba5846455094582097409f5923cffbfd not found: ID does not exist"
Oct 29 23:52:49.393269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.393234    5428 scope.go:122] "RemoveContainer" containerID="dc89684e7e7619f1c137f31d6f861fc3db539886352a9ee2f2f431f0522cb77d"
Oct 29 23:52:49.402985 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.402965    5428 scope.go:122] "RemoveContainer" containerID="aee97831fe33e6a456d2c3de5ad3b8878f14f57135fe327039c1d05ee651d359"
Oct 29 23:52:49.413279 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.413258    5428 scope.go:122] "RemoveContainer" containerID="dc89684e7e7619f1c137f31d6f861fc3db539886352a9ee2f2f431f0522cb77d"
Oct 29 23:52:49.413548 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:52:49.413529    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc89684e7e7619f1c137f31d6f861fc3db539886352a9ee2f2f431f0522cb77d\": container with ID starting with dc89684e7e7619f1c137f31d6f861fc3db539886352a9ee2f2f431f0522cb77d not found: ID does not exist" containerID="dc89684e7e7619f1c137f31d6f861fc3db539886352a9ee2f2f431f0522cb77d"
Oct 29 23:52:49.413604 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.413558    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc89684e7e7619f1c137f31d6f861fc3db539886352a9ee2f2f431f0522cb77d"} err="failed to get container status \"dc89684e7e7619f1c137f31d6f861fc3db539886352a9ee2f2f431f0522cb77d\": rpc error: code = NotFound desc = could not find container \"dc89684e7e7619f1c137f31d6f861fc3db539886352a9ee2f2f431f0522cb77d\": container with ID starting with dc89684e7e7619f1c137f31d6f861fc3db539886352a9ee2f2f431f0522cb77d not found: ID does not exist"
Oct 29 23:52:49.413604 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.413575    5428 scope.go:122] "RemoveContainer" containerID="aee97831fe33e6a456d2c3de5ad3b8878f14f57135fe327039c1d05ee651d359"
Oct 29 23:52:49.413795 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:52:49.413780    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aee97831fe33e6a456d2c3de5ad3b8878f14f57135fe327039c1d05ee651d359\": container with ID starting with aee97831fe33e6a456d2c3de5ad3b8878f14f57135fe327039c1d05ee651d359 not found: ID does not exist" containerID="aee97831fe33e6a456d2c3de5ad3b8878f14f57135fe327039c1d05ee651d359"
Oct 29 23:52:49.413848 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.413813    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee97831fe33e6a456d2c3de5ad3b8878f14f57135fe327039c1d05ee651d359"} err="failed to get container status \"aee97831fe33e6a456d2c3de5ad3b8878f14f57135fe327039c1d05ee651d359\": rpc error: code = NotFound desc = could not find container \"aee97831fe33e6a456d2c3de5ad3b8878f14f57135fe327039c1d05ee651d359\": container with ID starting with aee97831fe33e6a456d2c3de5ad3b8878f14f57135fe327039c1d05ee651d359 not found: ID does not exist"
Oct 29 23:52:49.440911 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.440884    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"]
Oct 29 23:52:49.444632 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.444609    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-jvn9h"]
Oct 29 23:52:49.478749 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.478612    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"]
Oct 29 23:52:49.481967 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.481944    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-ltnw2"]
Oct 29 23:52:49.515525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.515493    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"]
Oct 29 23:52:49.519409 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.519384    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-825472-openshift-default-6c98bc67c-xz8zr"]
Oct 29 23:52:49.825728 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.825603    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:49.950302 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:49.950269    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:50.146122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:50.146082    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"]
Oct 29 23:52:50.146632 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:50.146331    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm" podUID="c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" containerName="istio-proxy" containerID="cri-o://65a3081b7e301eadacc7f5de4f0b8136e2b3c84f5be1d8895eea76ec71a8b3c1" gracePeriod=30
Oct 29 23:52:50.176002 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:50.175964    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:50.776911 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:50.776876    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43dcb139-963e-4266-852b-d0119b2d7b0e" path="/var/lib/kubelet/pods/43dcb139-963e-4266-852b-d0119b2d7b0e/volumes"
Oct 29 23:52:50.777386 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:50.777372    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772a13d1-e842-4cf0-b36e-09e07d7936e4" path="/var/lib/kubelet/pods/772a13d1-e842-4cf0-b36e-09e07d7936e4/volumes"
Oct 29 23:52:50.777829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:50.777792    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" path="/var/lib/kubelet/pods/dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c/volumes"
Oct 29 23:52:51.103114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:51.103035    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:51.237647 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:51.237619    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:51.355408 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:51.355316    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:51.740367 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:51.740316    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:53.197286 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:53.197248    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:54.449114 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:54.449082    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:54.575502 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:54.575472    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:54.611396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:54.611103    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh"]
Oct 29 23:52:54.644653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:54.644617    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg"]
Oct 29 23:52:54.663032 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:54.663000    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh" podUID="e5d61a10-3f0c-42b8-a289-3c0830622387" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:52:54.755639 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:54.755555    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"]
Oct 29 23:52:54.755851 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:54.755826    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp" podUID="27603afe-5712-43b2-aaed-056d472de92a" containerName="istio-proxy" containerID="cri-o://0200d03b41b9ad520c31f614906adb40ce05e2382eb9ca23711e934ce154513b" gracePeriod=30
Oct 29 23:52:54.758128 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:54.758101    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:54.876950 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:54.876905    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:55.091704 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.091619    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:55.405614 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.405586    5428 generic.go:356] "Generic (PLEG): container finished" podID="c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" containerID="65a3081b7e301eadacc7f5de4f0b8136e2b3c84f5be1d8895eea76ec71a8b3c1" exitCode=0
Oct 29 23:52:55.405744 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.405655    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm" event={"ID":"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4","Type":"ContainerDied","Data":"65a3081b7e301eadacc7f5de4f0b8136e2b3c84f5be1d8895eea76ec71a8b3c1"}
Oct 29 23:52:55.405744 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.405704    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm" event={"ID":"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4","Type":"ContainerDied","Data":"4fad2e1d7205b2cd2a9abeace95be22d4201f29426a0ad7db9ef93c6be9b86e8"}
Oct 29 23:52:55.405744 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.405722    5428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fad2e1d7205b2cd2a9abeace95be22d4201f29426a0ad7db9ef93c6be9b86e8"
Oct 29 23:52:55.418766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.418749    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 23:52:55.480712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.480676    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-workload-socket\") pod \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") "
Oct 29 23:52:55.481249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.480729    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-token\" (UniqueName: \"kubernetes.io/projected/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-token\") pod \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") "
Oct 29 23:52:55.481249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.480766    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-credential-socket\") pod \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") "
Oct 29 23:52:55.481249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.480850    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-envoy\") pod \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") "
Oct 29 23:52:55.481249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.480883    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istiod-ca-cert\") pod \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") "
Oct 29 23:52:55.481249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.480950    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-kube-api-access-n8c4f\" (UniqueName: \"kubernetes.io/projected/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-kube-api-access-n8c4f\") pod \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") "
Oct 29 23:52:55.481249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.480961    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-workload-socket" pod "c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" (UID: "c7dec0d7-a62e-4d35-975b-cbe9bb9470e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:55.481249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.481032    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-workload-certs\") pod \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") "
Oct 29 23:52:55.481249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.481081    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-data\") pod \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") "
Oct 29 23:52:55.481249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.481126    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-podinfo\") pod \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\" (UID: \"c7dec0d7-a62e-4d35-975b-cbe9bb9470e4\") "
Oct 29 23:52:55.481791 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.481604    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:55.482903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.482650    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-credential-socket" pod "c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" (UID: "c7dec0d7-a62e-4d35-975b-cbe9bb9470e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:55.482903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.482668    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-data" pod "c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" (UID: "c7dec0d7-a62e-4d35-975b-cbe9bb9470e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:55.482903 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.482856    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istiod-ca-cert" pod "c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" (UID: "c7dec0d7-a62e-4d35-975b-cbe9bb9470e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:52:55.483467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.483432    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-workload-certs" pod "c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" (UID: "c7dec0d7-a62e-4d35-975b-cbe9bb9470e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:55.485144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.485113    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-kube-api-access-n8c4f" pod "c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" (UID: "c7dec0d7-a62e-4d35-975b-cbe9bb9470e4"). InnerVolumeSpecName "kube-api-access-n8c4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:52:55.486312 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.486280    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-podinfo" pod "c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" (UID: "c7dec0d7-a62e-4d35-975b-cbe9bb9470e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:52:55.486777 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.486754    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-envoy" pod "c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" (UID: "c7dec0d7-a62e-4d35-975b-cbe9bb9470e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:52:55.486900 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.486794    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-token" pod "c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" (UID: "c7dec0d7-a62e-4d35-975b-cbe9bb9470e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:52:55.583231 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.583198    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:55.583231 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.583227    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:55.583231 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.583237    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:55.583477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.583246    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:55.583477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.583255    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:55.583477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.583264    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:55.583477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.583277    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:55.583477 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:55.583286    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8c4f\" (UniqueName: \"kubernetes.io/projected/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4-kube-api-access-n8c4f\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:52:56.412248 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:56.412217    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"
Oct 29 23:52:56.476689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:56.476659    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"]
Oct 29 23:52:56.488457 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:56.488425    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1ee571-openshift-default-5c8565bfd9-cphkm"]
Oct 29 23:52:56.776744 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:56.776666    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" path="/var/lib/kubelet/pods/c7dec0d7-a62e-4d35-975b-cbe9bb9470e4/volumes"
Oct 29 23:52:56.858620 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:56.858591    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:58.038018 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:58.037986    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:58.166691 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:58.166660    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:58.412901 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:58.412869    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:58.711147 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:58.711062    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:52:59.612623 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:59.612550    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh" podUID="b1ac24ef-1667-4022-be72-55ffbd459242" containerName="payload-processing" containerID="cri-o://f4213bd68d0cd6abecaee77fe95a73a2c74008d8aad4455e01614be0603dde9f" gracePeriod=25
Oct 29 23:52:59.645195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:52:59.645137    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg" podUID="9205c9d1-7bc4-45ea-908f-776fb46a4bbe" containerName="payload-pre-processing" containerID="cri-o://97d23b71bd3f362e442a65041c8da631d9d398aa0dce723afbc28235646bbe65" gracePeriod=25
Oct 29 23:53:00.019210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.019177    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:53:00.130596 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.130522    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27603afe-5712-43b2-aaed-056d472de92a-istio-token\" (UniqueName: \"kubernetes.io/projected/27603afe-5712-43b2-aaed-056d472de92a-istio-token\") pod \"27603afe-5712-43b2-aaed-056d472de92a\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") "
Oct 29 23:53:00.130596 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.130568    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27603afe-5712-43b2-aaed-056d472de92a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27603afe-5712-43b2-aaed-056d472de92a-istio-podinfo\") pod \"27603afe-5712-43b2-aaed-056d472de92a\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") "
Oct 29 23:53:00.130883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.130645    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27603afe-5712-43b2-aaed-056d472de92a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27603afe-5712-43b2-aaed-056d472de92a-istiod-ca-cert\") pod \"27603afe-5712-43b2-aaed-056d472de92a\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") "
Oct 29 23:53:00.130883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.130697    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-workload-socket\") pod \"27603afe-5712-43b2-aaed-056d472de92a\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") "
Oct 29 23:53:00.130883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.130734    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-istio-data\") pod \"27603afe-5712-43b2-aaed-056d472de92a\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") "
Oct 29 23:53:00.130883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.130777    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-workload-certs\") pod \"27603afe-5712-43b2-aaed-056d472de92a\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") "
Oct 29 23:53:00.130883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.130833    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-credential-socket\") pod \"27603afe-5712-43b2-aaed-056d472de92a\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") "
Oct 29 23:53:00.131246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.130901    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-istio-envoy\") pod \"27603afe-5712-43b2-aaed-056d472de92a\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") "
Oct 29 23:53:00.131246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.130974    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27603afe-5712-43b2-aaed-056d472de92a-kube-api-access-47zlq\" (UniqueName: \"kubernetes.io/projected/27603afe-5712-43b2-aaed-056d472de92a-kube-api-access-47zlq\") pod \"27603afe-5712-43b2-aaed-056d472de92a\" (UID: \"27603afe-5712-43b2-aaed-056d472de92a\") "
Oct 29 23:53:00.131371 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.131284    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-istio-data" pod "27603afe-5712-43b2-aaed-056d472de92a" (UID: "27603afe-5712-43b2-aaed-056d472de92a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:53:00.131426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.131403    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27603afe-5712-43b2-aaed-056d472de92a-istiod-ca-cert" pod "27603afe-5712-43b2-aaed-056d472de92a" (UID: "27603afe-5712-43b2-aaed-056d472de92a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:53:00.131493 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.131425    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:00.131686 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.131656    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-workload-certs" pod "27603afe-5712-43b2-aaed-056d472de92a" (UID: "27603afe-5712-43b2-aaed-056d472de92a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:53:00.131686 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.131670    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-credential-socket" pod "27603afe-5712-43b2-aaed-056d472de92a" (UID: "27603afe-5712-43b2-aaed-056d472de92a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:53:00.131873 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.131694    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-workload-socket" pod "27603afe-5712-43b2-aaed-056d472de92a" (UID: "27603afe-5712-43b2-aaed-056d472de92a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:53:00.133297 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.133260    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27603afe-5712-43b2-aaed-056d472de92a-istio-token" pod "27603afe-5712-43b2-aaed-056d472de92a" (UID: "27603afe-5712-43b2-aaed-056d472de92a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:53:00.133507 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.133484    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-istio-envoy" pod "27603afe-5712-43b2-aaed-056d472de92a" (UID: "27603afe-5712-43b2-aaed-056d472de92a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:53:00.133602 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.133583    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27603afe-5712-43b2-aaed-056d472de92a-istio-podinfo" pod "27603afe-5712-43b2-aaed-056d472de92a" (UID: "27603afe-5712-43b2-aaed-056d472de92a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:53:00.133709 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.133692    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27603afe-5712-43b2-aaed-056d472de92a-kube-api-access-47zlq" pod "27603afe-5712-43b2-aaed-056d472de92a" (UID: "27603afe-5712-43b2-aaed-056d472de92a"). InnerVolumeSpecName "kube-api-access-47zlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:53:00.232332 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.232300    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27603afe-5712-43b2-aaed-056d472de92a-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:00.232332 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.232330    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:00.232332 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.232339    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:00.232561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.232348    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:00.232561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.232358    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27603afe-5712-43b2-aaed-056d472de92a-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:00.232561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.232366    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47zlq\" (UniqueName: \"kubernetes.io/projected/27603afe-5712-43b2-aaed-056d472de92a-kube-api-access-47zlq\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:00.232561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.232374    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27603afe-5712-43b2-aaed-056d472de92a-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:00.232561 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.232382    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27603afe-5712-43b2-aaed-056d472de92a-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:00.442743 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.442663    5428 generic.go:356] "Generic (PLEG): container finished" podID="27603afe-5712-43b2-aaed-056d472de92a" containerID="0200d03b41b9ad520c31f614906adb40ce05e2382eb9ca23711e934ce154513b" exitCode=0
Oct 29 23:53:00.442743 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.442715    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"
Oct 29 23:53:00.442967 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.442747    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp" event={"ID":"27603afe-5712-43b2-aaed-056d472de92a","Type":"ContainerDied","Data":"0200d03b41b9ad520c31f614906adb40ce05e2382eb9ca23711e934ce154513b"}
Oct 29 23:53:00.442967 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.442854    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp" event={"ID":"27603afe-5712-43b2-aaed-056d472de92a","Type":"ContainerDied","Data":"29077d5a405fdd27105c16bd47b038607034045568dcb95df9301c12f92b027b"}
Oct 29 23:53:00.442967 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.442885    5428 scope.go:122] "RemoveContainer" containerID="0200d03b41b9ad520c31f614906adb40ce05e2382eb9ca23711e934ce154513b"
Oct 29 23:53:00.454046 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.454028    5428 scope.go:122] "RemoveContainer" containerID="d4637797b8d50243bb72281afd00d1ca22f09fcb4e67ceda6cc7252bbd1c49f7"
Oct 29 23:53:00.463963 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.463945    5428 scope.go:122] "RemoveContainer" containerID="0200d03b41b9ad520c31f614906adb40ce05e2382eb9ca23711e934ce154513b"
Oct 29 23:53:00.464220 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:53:00.464203    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0200d03b41b9ad520c31f614906adb40ce05e2382eb9ca23711e934ce154513b\": container with ID starting with 0200d03b41b9ad520c31f614906adb40ce05e2382eb9ca23711e934ce154513b not found: ID does not exist" containerID="0200d03b41b9ad520c31f614906adb40ce05e2382eb9ca23711e934ce154513b"
Oct 29 23:53:00.464288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.464229    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0200d03b41b9ad520c31f614906adb40ce05e2382eb9ca23711e934ce154513b"} err="failed to get container status \"0200d03b41b9ad520c31f614906adb40ce05e2382eb9ca23711e934ce154513b\": rpc error: code = NotFound desc = could not find container \"0200d03b41b9ad520c31f614906adb40ce05e2382eb9ca23711e934ce154513b\": container with ID starting with 0200d03b41b9ad520c31f614906adb40ce05e2382eb9ca23711e934ce154513b not found: ID does not exist"
Oct 29 23:53:00.464288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.464244    5428 scope.go:122] "RemoveContainer" containerID="d4637797b8d50243bb72281afd00d1ca22f09fcb4e67ceda6cc7252bbd1c49f7"
Oct 29 23:53:00.464487 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:53:00.464466    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4637797b8d50243bb72281afd00d1ca22f09fcb4e67ceda6cc7252bbd1c49f7\": container with ID starting with d4637797b8d50243bb72281afd00d1ca22f09fcb4e67ceda6cc7252bbd1c49f7 not found: ID does not exist" containerID="d4637797b8d50243bb72281afd00d1ca22f09fcb4e67ceda6cc7252bbd1c49f7"
Oct 29 23:53:00.464539 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.464497    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4637797b8d50243bb72281afd00d1ca22f09fcb4e67ceda6cc7252bbd1c49f7"} err="failed to get container status \"d4637797b8d50243bb72281afd00d1ca22f09fcb4e67ceda6cc7252bbd1c49f7\": rpc error: code = NotFound desc = could not find container \"d4637797b8d50243bb72281afd00d1ca22f09fcb4e67ceda6cc7252bbd1c49f7\": container with ID starting with d4637797b8d50243bb72281afd00d1ca22f09fcb4e67ceda6cc7252bbd1c49f7 not found: ID does not exist"
Oct 29 23:53:00.495284 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.495253    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"]
Oct 29 23:53:00.509780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.509742    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-adc7d7-openshift-default-68549555b9-fdxtp"]
Oct 29 23:53:00.527409 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.527363    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:00.776796 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:00.776717    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27603afe-5712-43b2-aaed-056d472de92a" path="/var/lib/kubelet/pods/27603afe-5712-43b2-aaed-056d472de92a/volumes"
Oct 29 23:53:02.179597 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:02.179562    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:03.670193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:03.670159    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"]
Oct 29 23:53:03.670705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:03.670375    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2" podUID="d7c35813-8847-4178-9941-e6e5165078ac" containerName="maas-api" containerID="cri-o://1dfe94f2335cdc4870db82424ebe0ac3c1dde5a02618416ddf17143369085c1c" gracePeriod=30
Oct 29 23:53:03.739424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:03.739379    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:03.974113 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:03.974086    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:53:04.075587 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.075546    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7c35813-8847-4178-9941-e6e5165078ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c35813-8847-4178-9941-e6e5165078ac-openshift-service-ca\") pod \"d7c35813-8847-4178-9941-e6e5165078ac\" (UID: \"d7c35813-8847-4178-9941-e6e5165078ac\") "
Oct 29 23:53:04.075816 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.075693    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7c35813-8847-4178-9941-e6e5165078ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7c35813-8847-4178-9941-e6e5165078ac-maas-api-tls\") pod \"d7c35813-8847-4178-9941-e6e5165078ac\" (UID: \"d7c35813-8847-4178-9941-e6e5165078ac\") "
Oct 29 23:53:04.075816 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.075766    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7c35813-8847-4178-9941-e6e5165078ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c35813-8847-4178-9941-e6e5165078ac-metrics-certs\") pod \"d7c35813-8847-4178-9941-e6e5165078ac\" (UID: \"d7c35813-8847-4178-9941-e6e5165078ac\") "
Oct 29 23:53:04.075974 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.075853    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c35813-8847-4178-9941-e6e5165078ac-kube-api-access-xn6jm\" (UniqueName: \"kubernetes.io/projected/d7c35813-8847-4178-9941-e6e5165078ac-kube-api-access-xn6jm\") pod \"d7c35813-8847-4178-9941-e6e5165078ac\" (UID: \"d7c35813-8847-4178-9941-e6e5165078ac\") "
Oct 29 23:53:04.076069 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.076043    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c35813-8847-4178-9941-e6e5165078ac-openshift-service-ca" pod "d7c35813-8847-4178-9941-e6e5165078ac" (UID: "d7c35813-8847-4178-9941-e6e5165078ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:53:04.076388 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.076360    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c35813-8847-4178-9941-e6e5165078ac-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:04.078075 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.078044    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c35813-8847-4178-9941-e6e5165078ac-metrics-certs" pod "d7c35813-8847-4178-9941-e6e5165078ac" (UID: "d7c35813-8847-4178-9941-e6e5165078ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:53:04.078360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.078323    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c35813-8847-4178-9941-e6e5165078ac-maas-api-tls" pod "d7c35813-8847-4178-9941-e6e5165078ac" (UID: "d7c35813-8847-4178-9941-e6e5165078ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:53:04.078360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.078348    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c35813-8847-4178-9941-e6e5165078ac-kube-api-access-xn6jm" pod "d7c35813-8847-4178-9941-e6e5165078ac" (UID: "d7c35813-8847-4178-9941-e6e5165078ac"). InnerVolumeSpecName "kube-api-access-xn6jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:53:04.178823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.177997    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c35813-8847-4178-9941-e6e5165078ac-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:04.178823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.178026    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn6jm\" (UniqueName: \"kubernetes.io/projected/d7c35813-8847-4178-9941-e6e5165078ac-kube-api-access-xn6jm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:04.178823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.178040    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7c35813-8847-4178-9941-e6e5165078ac-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:04.330661 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.330579    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:04.468362 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.468327    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:04.474852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.474818    5428 generic.go:356] "Generic (PLEG): container finished" podID="d7c35813-8847-4178-9941-e6e5165078ac" containerID="1dfe94f2335cdc4870db82424ebe0ac3c1dde5a02618416ddf17143369085c1c" exitCode=0
Oct 29 23:53:04.474994 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.474886    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"
Oct 29 23:53:04.474994 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.474906    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2" event={"ID":"d7c35813-8847-4178-9941-e6e5165078ac","Type":"ContainerDied","Data":"1dfe94f2335cdc4870db82424ebe0ac3c1dde5a02618416ddf17143369085c1c"}
Oct 29 23:53:04.474994 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.474988    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2" event={"ID":"d7c35813-8847-4178-9941-e6e5165078ac","Type":"ContainerDied","Data":"4eb570f8fdfc1bf590c020e0a5a896854827c2c596b2384d922e7b90b3ab88d6"}
Oct 29 23:53:04.475129 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.475017    5428 scope.go:122] "RemoveContainer" containerID="1dfe94f2335cdc4870db82424ebe0ac3c1dde5a02618416ddf17143369085c1c"
Oct 29 23:53:04.487797 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.487777    5428 scope.go:122] "RemoveContainer" containerID="1dfe94f2335cdc4870db82424ebe0ac3c1dde5a02618416ddf17143369085c1c"
Oct 29 23:53:04.488104 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:53:04.488087    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dfe94f2335cdc4870db82424ebe0ac3c1dde5a02618416ddf17143369085c1c\": container with ID starting with 1dfe94f2335cdc4870db82424ebe0ac3c1dde5a02618416ddf17143369085c1c not found: ID does not exist" containerID="1dfe94f2335cdc4870db82424ebe0ac3c1dde5a02618416ddf17143369085c1c"
Oct 29 23:53:04.488155 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.488110    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfe94f2335cdc4870db82424ebe0ac3c1dde5a02618416ddf17143369085c1c"} err="failed to get container status \"1dfe94f2335cdc4870db82424ebe0ac3c1dde5a02618416ddf17143369085c1c\": rpc error: code = NotFound desc = could not find container \"1dfe94f2335cdc4870db82424ebe0ac3c1dde5a02618416ddf17143369085c1c\": container with ID starting with 1dfe94f2335cdc4870db82424ebe0ac3c1dde5a02618416ddf17143369085c1c not found: ID does not exist"
Oct 29 23:53:04.528766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.528732    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"]
Oct 29 23:53:04.532126 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.532088    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-45036b-6cffb9f64d-2r2c2"]
Oct 29 23:53:04.587671 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.587585    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:04.663699 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.663657    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh" podUID="e5d61a10-3f0c-42b8-a289-3c0830622387" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:53:04.776701 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.776669    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7c35813-8847-4178-9941-e6e5165078ac" path="/var/lib/kubelet/pods/d7c35813-8847-4178-9941-e6e5165078ac/volumes"
Oct 29 23:53:04.969428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:04.969400    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh"
Oct 29 23:53:05.087688 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:05.087652    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5d61a10-3f0c-42b8-a289-3c0830622387-kube-api-access-gh6cz\" (UniqueName: \"kubernetes.io/projected/e5d61a10-3f0c-42b8-a289-3c0830622387-kube-api-access-gh6cz\") pod \"e5d61a10-3f0c-42b8-a289-3c0830622387\" (UID: \"e5d61a10-3f0c-42b8-a289-3c0830622387\") "
Oct 29 23:53:05.087927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:05.087731    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5d61a10-3f0c-42b8-a289-3c0830622387-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d61a10-3f0c-42b8-a289-3c0830622387-plugins-config-volume\") pod \"e5d61a10-3f0c-42b8-a289-3c0830622387\" (UID: \"e5d61a10-3f0c-42b8-a289-3c0830622387\") "
Oct 29 23:53:05.088190 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:05.088163    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d61a10-3f0c-42b8-a289-3c0830622387-plugins-config-volume" pod "e5d61a10-3f0c-42b8-a289-3c0830622387" (UID: "e5d61a10-3f0c-42b8-a289-3c0830622387"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:53:05.089886 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:05.089860    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d61a10-3f0c-42b8-a289-3c0830622387-kube-api-access-gh6cz" pod "e5d61a10-3f0c-42b8-a289-3c0830622387" (UID: "e5d61a10-3f0c-42b8-a289-3c0830622387"). InnerVolumeSpecName "kube-api-access-gh6cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:53:05.190249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:05.190169    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh6cz\" (UniqueName: \"kubernetes.io/projected/e5d61a10-3f0c-42b8-a289-3c0830622387-kube-api-access-gh6cz\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:05.190249 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:05.190197    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d61a10-3f0c-42b8-a289-3c0830622387-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:05.483564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:05.483473    5428 generic.go:356] "Generic (PLEG): container finished" podID="e5d61a10-3f0c-42b8-a289-3c0830622387" containerID="ec1eb6dabcd3d09d128fde48fea76ac9a9430911954c5ecbc3a7f921aade5d4f" exitCode=0
Oct 29 23:53:05.483564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:05.483520    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh"
Oct 29 23:53:05.483564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:05.483550    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh" event={"ID":"e5d61a10-3f0c-42b8-a289-3c0830622387","Type":"ContainerDied","Data":"ec1eb6dabcd3d09d128fde48fea76ac9a9430911954c5ecbc3a7f921aade5d4f"}
Oct 29 23:53:05.483868 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:05.483596    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh" event={"ID":"e5d61a10-3f0c-42b8-a289-3c0830622387","Type":"ContainerDied","Data":"ea90e1ef50065d463a35c5b29f669bae31c33ce59b7943ca4cd843c3c3cdeca9"}
Oct 29 23:53:05.483868 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:05.483619    5428 scope.go:122] "RemoveContainer" containerID="ec1eb6dabcd3d09d128fde48fea76ac9a9430911954c5ecbc3a7f921aade5d4f"
Oct 29 23:53:05.494961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:05.494943    5428 scope.go:122] "RemoveContainer" containerID="ec1eb6dabcd3d09d128fde48fea76ac9a9430911954c5ecbc3a7f921aade5d4f"
Oct 29 23:53:05.495226 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:53:05.495206    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1eb6dabcd3d09d128fde48fea76ac9a9430911954c5ecbc3a7f921aade5d4f\": container with ID starting with ec1eb6dabcd3d09d128fde48fea76ac9a9430911954c5ecbc3a7f921aade5d4f not found: ID does not exist" containerID="ec1eb6dabcd3d09d128fde48fea76ac9a9430911954c5ecbc3a7f921aade5d4f"
Oct 29 23:53:05.495303 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:05.495239    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1eb6dabcd3d09d128fde48fea76ac9a9430911954c5ecbc3a7f921aade5d4f"} err="failed to get container status \"ec1eb6dabcd3d09d128fde48fea76ac9a9430911954c5ecbc3a7f921aade5d4f\": rpc error: code = NotFound desc = could not find container \"ec1eb6dabcd3d09d128fde48fea76ac9a9430911954c5ecbc3a7f921aade5d4f\": container with ID starting with ec1eb6dabcd3d09d128fde48fea76ac9a9430911954c5ecbc3a7f921aade5d4f not found: ID does not exist"
Oct 29 23:53:05.536819 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:05.536777    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh"]
Oct 29 23:53:05.548281 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:05.548257    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-adc7d7-659f9f6f79-cdpqh"]
Oct 29 23:53:06.491353 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:06.491308    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:06.776689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:06.776618    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5d61a10-3f0c-42b8-a289-3c0830622387" path="/var/lib/kubelet/pods/e5d61a10-3f0c-42b8-a289-3c0830622387/volumes"
Oct 29 23:53:08.294092 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:08.294059    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:09.019356 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:09.019313    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg" podUID="9205c9d1-7bc4-45ea-908f-776fb46a4bbe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:53:09.019583 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:09.019420    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh" podUID="b1ac24ef-1667-4022-be72-55ffbd459242" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:53:09.392286 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:09.392256    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj"]
Oct 29 23:53:09.507788 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:09.507750    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:09.620167 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:09.620128    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:10.131119 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:10.131090    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:11.853096 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:11.853057    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:13.631481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:13.631414    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:14.393363 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:14.393322    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj" podUID="999d9e5a-1f9f-4204-ba58-18317566426b" containerName="payload-pre-processing" containerID="cri-o://fd11fd86f1b9c68a355a186e76358a1e6444b0804611f9a3a3eb4bf98bc8a019" gracePeriod=25
Oct 29 23:53:14.546720 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:14.546689    5428 generic.go:356] "Generic (PLEG): container finished" podID="999d9e5a-1f9f-4204-ba58-18317566426b" containerID="fd11fd86f1b9c68a355a186e76358a1e6444b0804611f9a3a3eb4bf98bc8a019" exitCode=0
Oct 29 23:53:14.546887 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:14.546756    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj" event={"ID":"999d9e5a-1f9f-4204-ba58-18317566426b","Type":"ContainerDied","Data":"fd11fd86f1b9c68a355a186e76358a1e6444b0804611f9a3a3eb4bf98bc8a019"}
Oct 29 23:53:14.662934 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:14.662910    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj"
Oct 29 23:53:14.817843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:14.817787    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999d9e5a-1f9f-4204-ba58-18317566426b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999d9e5a-1f9f-4204-ba58-18317566426b-plugins-config-volume\") pod \"999d9e5a-1f9f-4204-ba58-18317566426b\" (UID: \"999d9e5a-1f9f-4204-ba58-18317566426b\") "
Oct 29 23:53:14.817999 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:14.817870    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999d9e5a-1f9f-4204-ba58-18317566426b-kube-api-access-r7txk\" (UniqueName: \"kubernetes.io/projected/999d9e5a-1f9f-4204-ba58-18317566426b-kube-api-access-r7txk\") pod \"999d9e5a-1f9f-4204-ba58-18317566426b\" (UID: \"999d9e5a-1f9f-4204-ba58-18317566426b\") "
Oct 29 23:53:14.818181 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:14.818151    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999d9e5a-1f9f-4204-ba58-18317566426b-plugins-config-volume" pod "999d9e5a-1f9f-4204-ba58-18317566426b" (UID: "999d9e5a-1f9f-4204-ba58-18317566426b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:53:14.818269 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:14.818254    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999d9e5a-1f9f-4204-ba58-18317566426b-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:14.819972 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:14.819952    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999d9e5a-1f9f-4204-ba58-18317566426b-kube-api-access-r7txk" pod "999d9e5a-1f9f-4204-ba58-18317566426b" (UID: "999d9e5a-1f9f-4204-ba58-18317566426b"). InnerVolumeSpecName "kube-api-access-r7txk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:53:14.918971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:14.918890    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7txk\" (UniqueName: \"kubernetes.io/projected/999d9e5a-1f9f-4204-ba58-18317566426b-kube-api-access-r7txk\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:14.973866 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:14.973828    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:15.007593 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.007559    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"]
Oct 29 23:53:15.007828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.007783    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq" podUID="fa67df40-c3c5-4e47-b3cb-bc5780382c69" containerName="maas-api" containerID="cri-o://4fa9a55abf57a0bce709381b2b6695b6323ca9fc4643c8fe2a92e9eb67120b93" gracePeriod=30
Oct 29 23:53:15.087856 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.087823    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:15.266663 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.266625    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"
Oct 29 23:53:15.424774 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.424745    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa67df40-c3c5-4e47-b3cb-bc5780382c69-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa67df40-c3c5-4e47-b3cb-bc5780382c69-metrics-certs\") pod \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\" (UID: \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\") "
Oct 29 23:53:15.424957 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.424819    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa67df40-c3c5-4e47-b3cb-bc5780382c69-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa67df40-c3c5-4e47-b3cb-bc5780382c69-openshift-service-ca\") pod \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\" (UID: \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\") "
Oct 29 23:53:15.424957 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.424871    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa67df40-c3c5-4e47-b3cb-bc5780382c69-kube-api-access-bklq5\" (UniqueName: \"kubernetes.io/projected/fa67df40-c3c5-4e47-b3cb-bc5780382c69-kube-api-access-bklq5\") pod \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\" (UID: \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\") "
Oct 29 23:53:15.425085 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.424983    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa67df40-c3c5-4e47-b3cb-bc5780382c69-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa67df40-c3c5-4e47-b3cb-bc5780382c69-maas-api-tls\") pod \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\" (UID: \"fa67df40-c3c5-4e47-b3cb-bc5780382c69\") "
Oct 29 23:53:15.425208 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.425185    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa67df40-c3c5-4e47-b3cb-bc5780382c69-openshift-service-ca" pod "fa67df40-c3c5-4e47-b3cb-bc5780382c69" (UID: "fa67df40-c3c5-4e47-b3cb-bc5780382c69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:53:15.425568 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.425547    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa67df40-c3c5-4e47-b3cb-bc5780382c69-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:15.427143 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.427118    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa67df40-c3c5-4e47-b3cb-bc5780382c69-metrics-certs" pod "fa67df40-c3c5-4e47-b3cb-bc5780382c69" (UID: "fa67df40-c3c5-4e47-b3cb-bc5780382c69"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:53:15.427272 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.427240    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa67df40-c3c5-4e47-b3cb-bc5780382c69-kube-api-access-bklq5" pod "fa67df40-c3c5-4e47-b3cb-bc5780382c69" (UID: "fa67df40-c3c5-4e47-b3cb-bc5780382c69"). InnerVolumeSpecName "kube-api-access-bklq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:53:15.427321 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.427288    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa67df40-c3c5-4e47-b3cb-bc5780382c69-maas-api-tls" pod "fa67df40-c3c5-4e47-b3cb-bc5780382c69" (UID: "fa67df40-c3c5-4e47-b3cb-bc5780382c69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:53:15.527652 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.527573    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa67df40-c3c5-4e47-b3cb-bc5780382c69-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:15.527857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.527675    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bklq5\" (UniqueName: \"kubernetes.io/projected/fa67df40-c3c5-4e47-b3cb-bc5780382c69-kube-api-access-bklq5\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:15.527857 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.527703    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa67df40-c3c5-4e47-b3cb-bc5780382c69-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:15.553883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.553851    5428 generic.go:356] "Generic (PLEG): container finished" podID="fa67df40-c3c5-4e47-b3cb-bc5780382c69" containerID="4fa9a55abf57a0bce709381b2b6695b6323ca9fc4643c8fe2a92e9eb67120b93" exitCode=0
Oct 29 23:53:15.554047 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.553902    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"
Oct 29 23:53:15.554047 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.553940    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq" event={"ID":"fa67df40-c3c5-4e47-b3cb-bc5780382c69","Type":"ContainerDied","Data":"4fa9a55abf57a0bce709381b2b6695b6323ca9fc4643c8fe2a92e9eb67120b93"}
Oct 29 23:53:15.554047 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.554019    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq" event={"ID":"fa67df40-c3c5-4e47-b3cb-bc5780382c69","Type":"ContainerDied","Data":"cf5fadae66f282580ad7380f88ec698792e1d5052a8a2d6b83110ac526654a4e"}
Oct 29 23:53:15.554200 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.554051    5428 scope.go:122] "RemoveContainer" containerID="4fa9a55abf57a0bce709381b2b6695b6323ca9fc4643c8fe2a92e9eb67120b93"
Oct 29 23:53:15.555541 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.555403    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj" event={"ID":"999d9e5a-1f9f-4204-ba58-18317566426b","Type":"ContainerDied","Data":"1f9f4f2f3b3ab6bda11770f36e94e069d0dabe14d6989f76246fd31e84f920a2"}
Oct 29 23:53:15.555541 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.555423    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj"
Oct 29 23:53:15.565975 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.565956    5428 scope.go:122] "RemoveContainer" containerID="4fa9a55abf57a0bce709381b2b6695b6323ca9fc4643c8fe2a92e9eb67120b93"
Oct 29 23:53:15.566258 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:53:15.566239    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa9a55abf57a0bce709381b2b6695b6323ca9fc4643c8fe2a92e9eb67120b93\": container with ID starting with 4fa9a55abf57a0bce709381b2b6695b6323ca9fc4643c8fe2a92e9eb67120b93 not found: ID does not exist" containerID="4fa9a55abf57a0bce709381b2b6695b6323ca9fc4643c8fe2a92e9eb67120b93"
Oct 29 23:53:15.566337 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.566264    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa9a55abf57a0bce709381b2b6695b6323ca9fc4643c8fe2a92e9eb67120b93"} err="failed to get container status \"4fa9a55abf57a0bce709381b2b6695b6323ca9fc4643c8fe2a92e9eb67120b93\": rpc error: code = NotFound desc = could not find container \"4fa9a55abf57a0bce709381b2b6695b6323ca9fc4643c8fe2a92e9eb67120b93\": container with ID starting with 4fa9a55abf57a0bce709381b2b6695b6323ca9fc4643c8fe2a92e9eb67120b93 not found: ID does not exist"
Oct 29 23:53:15.566337 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.566285    5428 scope.go:122] "RemoveContainer" containerID="fd11fd86f1b9c68a355a186e76358a1e6444b0804611f9a3a3eb4bf98bc8a019"
Oct 29 23:53:15.605278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.605250    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj"]
Oct 29 23:53:15.610347 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.610324    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-a4f2db-7b6fb574f9-fl6cj"]
Oct 29 23:53:15.641636 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.641609    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"]
Oct 29 23:53:15.645786 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.645763    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-a4f2db-7d54c56948-87wfq"]
Oct 29 23:53:15.697451 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.697367    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:15.828956 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.828903    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:15.947195 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:15.947161    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:16.313338 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:16.313308    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:16.776607 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:16.776570    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="999d9e5a-1f9f-4204-ba58-18317566426b" path="/var/lib/kubelet/pods/999d9e5a-1f9f-4204-ba58-18317566426b/volumes"
Oct 29 23:53:16.777194 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:16.777172    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa67df40-c3c5-4e47-b3cb-bc5780382c69" path="/var/lib/kubelet/pods/fa67df40-c3c5-4e47-b3cb-bc5780382c69/volumes"
Oct 29 23:53:17.367876 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:53:17.367832    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:53:17.702525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:17.702437    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:19.019239 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:19.019201    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh" podUID="b1ac24ef-1667-4022-be72-55ffbd459242" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:53:19.019663 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:19.019200    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg" podUID="9205c9d1-7bc4-45ea-908f-776fb46a4bbe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:53:19.390670 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:19.390639    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:19.903261 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:19.903228    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh"
Oct 29 23:53:19.965111 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:19.965046    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg"
Oct 29 23:53:19.977011 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:19.976984    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1ac24ef-1667-4022-be72-55ffbd459242-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ac24ef-1667-4022-be72-55ffbd459242-plugins-config-volume\") pod \"b1ac24ef-1667-4022-be72-55ffbd459242\" (UID: \"b1ac24ef-1667-4022-be72-55ffbd459242\") "
Oct 29 23:53:19.977150 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:19.977035    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ac24ef-1667-4022-be72-55ffbd459242-kube-api-access-df4wl\" (UniqueName: \"kubernetes.io/projected/b1ac24ef-1667-4022-be72-55ffbd459242-kube-api-access-df4wl\") pod \"b1ac24ef-1667-4022-be72-55ffbd459242\" (UID: \"b1ac24ef-1667-4022-be72-55ffbd459242\") "
Oct 29 23:53:19.977323 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:19.977301    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ac24ef-1667-4022-be72-55ffbd459242-plugins-config-volume" pod "b1ac24ef-1667-4022-be72-55ffbd459242" (UID: "b1ac24ef-1667-4022-be72-55ffbd459242"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:53:19.977481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:19.977467    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ac24ef-1667-4022-be72-55ffbd459242-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:19.979108 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:19.979084    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ac24ef-1667-4022-be72-55ffbd459242-kube-api-access-df4wl" pod "b1ac24ef-1667-4022-be72-55ffbd459242" (UID: "b1ac24ef-1667-4022-be72-55ffbd459242"). InnerVolumeSpecName "kube-api-access-df4wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:53:20.077917 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.077883    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9205c9d1-7bc4-45ea-908f-776fb46a4bbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9205c9d1-7bc4-45ea-908f-776fb46a4bbe-plugins-config-volume\") pod \"9205c9d1-7bc4-45ea-908f-776fb46a4bbe\" (UID: \"9205c9d1-7bc4-45ea-908f-776fb46a4bbe\") "
Oct 29 23:53:20.078327 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.077947    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9205c9d1-7bc4-45ea-908f-776fb46a4bbe-kube-api-access-5s7ws\" (UniqueName: \"kubernetes.io/projected/9205c9d1-7bc4-45ea-908f-776fb46a4bbe-kube-api-access-5s7ws\") pod \"9205c9d1-7bc4-45ea-908f-776fb46a4bbe\" (UID: \"9205c9d1-7bc4-45ea-908f-776fb46a4bbe\") "
Oct 29 23:53:20.078327 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.078242    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df4wl\" (UniqueName: \"kubernetes.io/projected/b1ac24ef-1667-4022-be72-55ffbd459242-kube-api-access-df4wl\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:20.078327 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.078243    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9205c9d1-7bc4-45ea-908f-776fb46a4bbe-plugins-config-volume" pod "9205c9d1-7bc4-45ea-908f-776fb46a4bbe" (UID: "9205c9d1-7bc4-45ea-908f-776fb46a4bbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:53:20.080122 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.080102    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9205c9d1-7bc4-45ea-908f-776fb46a4bbe-kube-api-access-5s7ws" pod "9205c9d1-7bc4-45ea-908f-776fb46a4bbe" (UID: "9205c9d1-7bc4-45ea-908f-776fb46a4bbe"). InnerVolumeSpecName "kube-api-access-5s7ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:53:20.178877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.178846    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9205c9d1-7bc4-45ea-908f-776fb46a4bbe-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:20.178877 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.178873    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s7ws\" (UniqueName: \"kubernetes.io/projected/9205c9d1-7bc4-45ea-908f-776fb46a4bbe-kube-api-access-5s7ws\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:20.588950 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.588915    5428 generic.go:356] "Generic (PLEG): container finished" podID="9205c9d1-7bc4-45ea-908f-776fb46a4bbe" containerID="97d23b71bd3f362e442a65041c8da631d9d398aa0dce723afbc28235646bbe65" exitCode=0
Oct 29 23:53:20.589148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.588962    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg"
Oct 29 23:53:20.589148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.589004    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg" event={"ID":"9205c9d1-7bc4-45ea-908f-776fb46a4bbe","Type":"ContainerDied","Data":"97d23b71bd3f362e442a65041c8da631d9d398aa0dce723afbc28235646bbe65"}
Oct 29 23:53:20.589148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.589091    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg" event={"ID":"9205c9d1-7bc4-45ea-908f-776fb46a4bbe","Type":"ContainerDied","Data":"87c39fe7bfc69ac43520d0e2bc3fb7f83533b254f76f08024bb159b96e901472"}
Oct 29 23:53:20.589148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.589119    5428 scope.go:122] "RemoveContainer" containerID="97d23b71bd3f362e442a65041c8da631d9d398aa0dce723afbc28235646bbe65"
Oct 29 23:53:20.590410 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.590385    5428 generic.go:356] "Generic (PLEG): container finished" podID="b1ac24ef-1667-4022-be72-55ffbd459242" containerID="f4213bd68d0cd6abecaee77fe95a73a2c74008d8aad4455e01614be0603dde9f" exitCode=0
Oct 29 23:53:20.590527 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.590433    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh" event={"ID":"b1ac24ef-1667-4022-be72-55ffbd459242","Type":"ContainerDied","Data":"f4213bd68d0cd6abecaee77fe95a73a2c74008d8aad4455e01614be0603dde9f"}
Oct 29 23:53:20.590527 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.590455    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh"
Oct 29 23:53:20.590643 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.590462    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh" event={"ID":"b1ac24ef-1667-4022-be72-55ffbd459242","Type":"ContainerDied","Data":"40e27bddb35bc789d75121389c01e330096e6fab1cc6ff55b06bb5b548379099"}
Oct 29 23:53:20.601718 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.601701    5428 scope.go:122] "RemoveContainer" containerID="97d23b71bd3f362e442a65041c8da631d9d398aa0dce723afbc28235646bbe65"
Oct 29 23:53:20.602013 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:53:20.601993    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d23b71bd3f362e442a65041c8da631d9d398aa0dce723afbc28235646bbe65\": container with ID starting with 97d23b71bd3f362e442a65041c8da631d9d398aa0dce723afbc28235646bbe65 not found: ID does not exist" containerID="97d23b71bd3f362e442a65041c8da631d9d398aa0dce723afbc28235646bbe65"
Oct 29 23:53:20.602073 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.602021    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d23b71bd3f362e442a65041c8da631d9d398aa0dce723afbc28235646bbe65"} err="failed to get container status \"97d23b71bd3f362e442a65041c8da631d9d398aa0dce723afbc28235646bbe65\": rpc error: code = NotFound desc = could not find container \"97d23b71bd3f362e442a65041c8da631d9d398aa0dce723afbc28235646bbe65\": container with ID starting with 97d23b71bd3f362e442a65041c8da631d9d398aa0dce723afbc28235646bbe65 not found: ID does not exist"
Oct 29 23:53:20.602073 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.602039    5428 scope.go:122] "RemoveContainer" containerID="f4213bd68d0cd6abecaee77fe95a73a2c74008d8aad4455e01614be0603dde9f"
Oct 29 23:53:20.611667 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.611644    5428 scope.go:122] "RemoveContainer" containerID="f4213bd68d0cd6abecaee77fe95a73a2c74008d8aad4455e01614be0603dde9f"
Oct 29 23:53:20.612064 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:53:20.612043    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4213bd68d0cd6abecaee77fe95a73a2c74008d8aad4455e01614be0603dde9f\": container with ID starting with f4213bd68d0cd6abecaee77fe95a73a2c74008d8aad4455e01614be0603dde9f not found: ID does not exist" containerID="f4213bd68d0cd6abecaee77fe95a73a2c74008d8aad4455e01614be0603dde9f"
Oct 29 23:53:20.612188 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.612069    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4213bd68d0cd6abecaee77fe95a73a2c74008d8aad4455e01614be0603dde9f"} err="failed to get container status \"f4213bd68d0cd6abecaee77fe95a73a2c74008d8aad4455e01614be0603dde9f\": rpc error: code = NotFound desc = could not find container \"f4213bd68d0cd6abecaee77fe95a73a2c74008d8aad4455e01614be0603dde9f\": container with ID starting with f4213bd68d0cd6abecaee77fe95a73a2c74008d8aad4455e01614be0603dde9f not found: ID does not exist"
Oct 29 23:53:20.644780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.644757    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg"]
Oct 29 23:53:20.657504 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.657481    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-45036b-5dcdf44b67-7gngg"]
Oct 29 23:53:20.692908 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.692875    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh"]
Oct 29 23:53:20.698883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.698860    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-45036b-5b78d5cffb-gvhkh"]
Oct 29 23:53:20.776464 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.776433    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9205c9d1-7bc4-45ea-908f-776fb46a4bbe" path="/var/lib/kubelet/pods/9205c9d1-7bc4-45ea-908f-776fb46a4bbe/volumes"
Oct 29 23:53:20.776898 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.776884    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1ac24ef-1667-4022-be72-55ffbd459242" path="/var/lib/kubelet/pods/b1ac24ef-1667-4022-be72-55ffbd459242/volumes"
Oct 29 23:53:20.978110 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:20.978065    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:21.092511 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:21.092482    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:21.228013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:21.227975    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:21.339722 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:21.339638    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:22.670791 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:22.670753    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:22.783217 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:22.783182    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:22.827488 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:22.827457    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"]
Oct 29 23:53:22.827705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:22.827665    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp" podUID="64d16a34-4567-4586-a992-297386786fad" containerName="maas-api" containerID="cri-o://177fcfe0668586cd1ba36e2b2f15517ea5eb552b07d6077d65db0f6d7680039e" gracePeriod=30
Oct 29 23:53:22.905591 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:22.905554    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:23.026373 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.026338    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:23.091181 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.090867    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"
Oct 29 23:53:23.210085 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.210000    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d16a34-4567-4586-a992-297386786fad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d16a34-4567-4586-a992-297386786fad-openshift-service-ca\") pod \"64d16a34-4567-4586-a992-297386786fad\" (UID: \"64d16a34-4567-4586-a992-297386786fad\") "
Oct 29 23:53:23.210085 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.210063    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d16a34-4567-4586-a992-297386786fad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d16a34-4567-4586-a992-297386786fad-metrics-certs\") pod \"64d16a34-4567-4586-a992-297386786fad\" (UID: \"64d16a34-4567-4586-a992-297386786fad\") "
Oct 29 23:53:23.210348 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.210111    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d16a34-4567-4586-a992-297386786fad-kube-api-access-x9m8v\" (UniqueName: \"kubernetes.io/projected/64d16a34-4567-4586-a992-297386786fad-kube-api-access-x9m8v\") pod \"64d16a34-4567-4586-a992-297386786fad\" (UID: \"64d16a34-4567-4586-a992-297386786fad\") "
Oct 29 23:53:23.210348 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.210146    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d16a34-4567-4586-a992-297386786fad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d16a34-4567-4586-a992-297386786fad-maas-api-tls\") pod \"64d16a34-4567-4586-a992-297386786fad\" (UID: \"64d16a34-4567-4586-a992-297386786fad\") "
Oct 29 23:53:23.210465 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.210382    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d16a34-4567-4586-a992-297386786fad-openshift-service-ca" pod "64d16a34-4567-4586-a992-297386786fad" (UID: "64d16a34-4567-4586-a992-297386786fad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:53:23.210550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.210533    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d16a34-4567-4586-a992-297386786fad-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:23.212325 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.212297    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d16a34-4567-4586-a992-297386786fad-metrics-certs" pod "64d16a34-4567-4586-a992-297386786fad" (UID: "64d16a34-4567-4586-a992-297386786fad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:53:23.212325 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.212310    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d16a34-4567-4586-a992-297386786fad-kube-api-access-x9m8v" pod "64d16a34-4567-4586-a992-297386786fad" (UID: "64d16a34-4567-4586-a992-297386786fad"). InnerVolumeSpecName "kube-api-access-x9m8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:53:23.212528 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.212355    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d16a34-4567-4586-a992-297386786fad-maas-api-tls" pod "64d16a34-4567-4586-a992-297386786fad" (UID: "64d16a34-4567-4586-a992-297386786fad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:53:23.311222 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.311187    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d16a34-4567-4586-a992-297386786fad-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:23.311222 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.311216    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9m8v\" (UniqueName: \"kubernetes.io/projected/64d16a34-4567-4586-a992-297386786fad-kube-api-access-x9m8v\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:23.311222 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.311227    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d16a34-4567-4586-a992-297386786fad-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:23.612930 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.612844    5428 generic.go:356] "Generic (PLEG): container finished" podID="64d16a34-4567-4586-a992-297386786fad" containerID="177fcfe0668586cd1ba36e2b2f15517ea5eb552b07d6077d65db0f6d7680039e" exitCode=0
Oct 29 23:53:23.612930 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.612894    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"
Oct 29 23:53:23.612930 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.612899    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp" event={"ID":"64d16a34-4567-4586-a992-297386786fad","Type":"ContainerDied","Data":"177fcfe0668586cd1ba36e2b2f15517ea5eb552b07d6077d65db0f6d7680039e"}
Oct 29 23:53:23.613189 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.612944    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp" event={"ID":"64d16a34-4567-4586-a992-297386786fad","Type":"ContainerDied","Data":"24bb118b9749df735a74d14c148915da2827eefbd784d75f5c8ea1ed085545bc"}
Oct 29 23:53:23.613189 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.612966    5428 scope.go:122] "RemoveContainer" containerID="177fcfe0668586cd1ba36e2b2f15517ea5eb552b07d6077d65db0f6d7680039e"
Oct 29 23:53:23.624384 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.624363    5428 scope.go:122] "RemoveContainer" containerID="177fcfe0668586cd1ba36e2b2f15517ea5eb552b07d6077d65db0f6d7680039e"
Oct 29 23:53:23.624627 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:53:23.624611    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"177fcfe0668586cd1ba36e2b2f15517ea5eb552b07d6077d65db0f6d7680039e\": container with ID starting with 177fcfe0668586cd1ba36e2b2f15517ea5eb552b07d6077d65db0f6d7680039e not found: ID does not exist" containerID="177fcfe0668586cd1ba36e2b2f15517ea5eb552b07d6077d65db0f6d7680039e"
Oct 29 23:53:23.624689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.624634    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177fcfe0668586cd1ba36e2b2f15517ea5eb552b07d6077d65db0f6d7680039e"} err="failed to get container status \"177fcfe0668586cd1ba36e2b2f15517ea5eb552b07d6077d65db0f6d7680039e\": rpc error: code = NotFound desc = could not find container \"177fcfe0668586cd1ba36e2b2f15517ea5eb552b07d6077d65db0f6d7680039e\": container with ID starting with 177fcfe0668586cd1ba36e2b2f15517ea5eb552b07d6077d65db0f6d7680039e not found: ID does not exist"
Oct 29 23:53:23.649917 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.649889    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:23.663291 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.663267    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"]
Oct 29 23:53:23.675215 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.675194    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a84df8-cf9f99569-mn8zp"]
Oct 29 23:53:23.793640 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.793591    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:23.898775 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.898739    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq"]
Oct 29 23:53:23.899270 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899255    5428 cpu_manager.go:414] "containerMap: removing container" podUID="d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec" containerName="maas-api"
Oct 29 23:53:23.899317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899271    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec" containerName="maas-api"
Oct 29 23:53:23.899317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899283    5428 cpu_manager.go:414] "containerMap: removing container" podUID="d7c35813-8847-4178-9941-e6e5165078ac" containerName="maas-api"
Oct 29 23:53:23.899317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899288    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c35813-8847-4178-9941-e6e5165078ac" containerName="maas-api"
Oct 29 23:53:23.899317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899294    5428 cpu_manager.go:414] "containerMap: removing container" podUID="27603afe-5712-43b2-aaed-056d472de92a" containerName="istio-proxy"
Oct 29 23:53:23.899317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899299    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27603afe-5712-43b2-aaed-056d472de92a" containerName="istio-proxy"
Oct 29 23:53:23.899317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899307    5428 cpu_manager.go:414] "containerMap: removing container" podUID="13040ff3-bf4c-4e93-8823-1feb043a8eda" containerName="payload-pre-processing"
Oct 29 23:53:23.899317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899311    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13040ff3-bf4c-4e93-8823-1feb043a8eda" containerName="payload-pre-processing"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899322    5428 cpu_manager.go:414] "containerMap: removing container" podUID="dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899327    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899333    5428 cpu_manager.go:414] "containerMap: removing container" podUID="27603afe-5712-43b2-aaed-056d472de92a" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899337    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27603afe-5712-43b2-aaed-056d472de92a" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899346    5428 cpu_manager.go:414] "containerMap: removing container" podUID="97ec11c4-83fa-4ae5-94fa-35a14eed785f" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899350    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ec11c4-83fa-4ae5-94fa-35a14eed785f" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899357    5428 cpu_manager.go:414] "containerMap: removing container" podUID="c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899361    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899368    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b1ac24ef-1667-4022-be72-55ffbd459242" containerName="payload-processing"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899373    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ac24ef-1667-4022-be72-55ffbd459242" containerName="payload-processing"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899386    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9205c9d1-7bc4-45ea-908f-776fb46a4bbe" containerName="payload-pre-processing"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899391    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9205c9d1-7bc4-45ea-908f-776fb46a4bbe" containerName="payload-pre-processing"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899397    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b034cafa-0951-4854-899a-a9f03ebb7748" containerName="payload-processing"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899402    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b034cafa-0951-4854-899a-a9f03ebb7748" containerName="payload-processing"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899410    5428 cpu_manager.go:414] "containerMap: removing container" podUID="772a13d1-e842-4cf0-b36e-09e07d7936e4" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899414    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772a13d1-e842-4cf0-b36e-09e07d7936e4" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899423    5428 cpu_manager.go:414] "containerMap: removing container" podUID="772a13d1-e842-4cf0-b36e-09e07d7936e4" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899427    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772a13d1-e842-4cf0-b36e-09e07d7936e4" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899435    5428 cpu_manager.go:414] "containerMap: removing container" podUID="97ec11c4-83fa-4ae5-94fa-35a14eed785f" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899440    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ec11c4-83fa-4ae5-94fa-35a14eed785f" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899455    5428 cpu_manager.go:414] "containerMap: removing container" podUID="c84aca64-8103-4122-9899-158745522b85" containerName="main"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899461    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c84aca64-8103-4122-9899-158745522b85" containerName="main"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899469    5428 cpu_manager.go:414] "containerMap: removing container" podUID="43dcb139-963e-4266-852b-d0119b2d7b0e" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899475    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43dcb139-963e-4266-852b-d0119b2d7b0e" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899483    5428 cpu_manager.go:414] "containerMap: removing container" podUID="85e2d99a-031b-441f-b881-df20fba27bba" containerName="payload-processing"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899487    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e2d99a-031b-441f-b881-df20fba27bba" containerName="payload-processing"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899494    5428 cpu_manager.go:414] "containerMap: removing container" podUID="fa67df40-c3c5-4e47-b3cb-bc5780382c69" containerName="maas-api"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899498    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa67df40-c3c5-4e47-b3cb-bc5780382c69" containerName="maas-api"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899508    5428 cpu_manager.go:414] "containerMap: removing container" podUID="0d2f66a5-0f31-4820-9693-b4d87b6c4933" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899513    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2f66a5-0f31-4820-9693-b4d87b6c4933" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899518    5428 cpu_manager.go:414] "containerMap: removing container" podUID="64d16a34-4567-4586-a992-297386786fad" containerName="maas-api"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899523    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d16a34-4567-4586-a992-297386786fad" containerName="maas-api"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899529    5428 cpu_manager.go:414] "containerMap: removing container" podUID="8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899534    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" containerName="istio-proxy"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899540    5428 cpu_manager.go:414] "containerMap: removing container" podUID="999d9e5a-1f9f-4204-ba58-18317566426b" containerName="payload-pre-processing"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899545    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999d9e5a-1f9f-4204-ba58-18317566426b" containerName="payload-pre-processing"
Oct 29 23:53:23.899543 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899556    5428 cpu_manager.go:414] "containerMap: removing container" podUID="e5d61a10-3f0c-42b8-a289-3c0830622387" containerName="payload-pre-processing"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899561    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5d61a10-3f0c-42b8-a289-3c0830622387" containerName="payload-pre-processing"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899568    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b6d6288f-a859-4dee-9fc8-389435a62863" containerName="maas-api"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899572    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d6288f-a859-4dee-9fc8-389435a62863" containerName="maas-api"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899584    5428 cpu_manager.go:414] "containerMap: removing container" podUID="29b17d6e-4c9f-4067-95b6-124bc1849a60" containerName="istio-proxy"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899589    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b17d6e-4c9f-4067-95b6-124bc1849a60" containerName="istio-proxy"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899650    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b17d6e-4c9f-4067-95b6-124bc1849a60" containerName="istio-proxy"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899657    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59b3e5a-dbbb-47e5-b2b1-67b9ef669dec" containerName="maas-api"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899664    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c35813-8847-4178-9941-e6e5165078ac" containerName="maas-api"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899671    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="27603afe-5712-43b2-aaed-056d472de92a" containerName="istio-proxy"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899679    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="13040ff3-bf4c-4e93-8823-1feb043a8eda" containerName="payload-pre-processing"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899685    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd646a4a-2d7d-4a9a-a66d-cc2f9c62f88c" containerName="istio-proxy"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899689    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="27603afe-5712-43b2-aaed-056d472de92a" containerName="istio-proxy"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899696    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ec11c4-83fa-4ae5-94fa-35a14eed785f" containerName="istio-proxy"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899702    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7dec0d7-a62e-4d35-975b-cbe9bb9470e4" containerName="istio-proxy"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899707    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ac24ef-1667-4022-be72-55ffbd459242" containerName="payload-processing"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899714    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9205c9d1-7bc4-45ea-908f-776fb46a4bbe" containerName="payload-pre-processing"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899719    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b034cafa-0951-4854-899a-a9f03ebb7748" containerName="payload-processing"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899723    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="772a13d1-e842-4cf0-b36e-09e07d7936e4" containerName="istio-proxy"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899729    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="772a13d1-e842-4cf0-b36e-09e07d7936e4" containerName="istio-proxy"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899733    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ec11c4-83fa-4ae5-94fa-35a14eed785f" containerName="istio-proxy"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899738    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c84aca64-8103-4122-9899-158745522b85" containerName="main"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899744    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="43dcb139-963e-4266-852b-d0119b2d7b0e" containerName="istio-proxy"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899748    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e2d99a-031b-441f-b881-df20fba27bba" containerName="payload-processing"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899752    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa67df40-c3c5-4e47-b3cb-bc5780382c69" containerName="maas-api"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899757    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2f66a5-0f31-4820-9693-b4d87b6c4933" containerName="istio-proxy"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899763    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d16a34-4567-4586-a992-297386786fad" containerName="maas-api"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899768    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c4d02b4-5d2b-466b-a379-fcbdd26b8f32" containerName="istio-proxy"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899773    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="999d9e5a-1f9f-4204-ba58-18317566426b" containerName="payload-pre-processing"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899780    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5d61a10-3f0c-42b8-a289-3c0830622387" containerName="payload-pre-processing"
Oct 29 23:53:23.900705 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.899785    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d6288f-a859-4dee-9fc8-389435a62863" containerName="maas-api"
Oct 29 23:53:23.904215 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.904197    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq"
Oct 29 23:53:23.919479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.919381    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:53:23.919479 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.919456    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\""
Oct 29 23:53:23.919951 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.919931    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:23.921182 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:23.921159    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq"]
Oct 29 23:53:24.019643 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:24.019609    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc793e2-94d3-4a12-a4ea-aa262bf53319-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq\" (UID: \"4dc793e2-94d3-4a12-a4ea-aa262bf53319\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq"
Oct 29 23:53:24.019818 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:24.019662    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc793e2-94d3-4a12-a4ea-aa262bf53319-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq\" (UID: \"4dc793e2-94d3-4a12-a4ea-aa262bf53319\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq"
Oct 29 23:53:24.019818 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:24.019783    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn54r\" (UniqueName: \"kubernetes.io/projected/4dc793e2-94d3-4a12-a4ea-aa262bf53319-kube-api-access-rn54r\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq\" (UID: \"4dc793e2-94d3-4a12-a4ea-aa262bf53319\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq"
Oct 29 23:53:24.120692 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:24.120661    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn54r\" (UniqueName: \"kubernetes.io/projected/4dc793e2-94d3-4a12-a4ea-aa262bf53319-kube-api-access-rn54r\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq\" (UID: \"4dc793e2-94d3-4a12-a4ea-aa262bf53319\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq"
Oct 29 23:53:24.120908 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:24.120852    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc793e2-94d3-4a12-a4ea-aa262bf53319-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq\" (UID: \"4dc793e2-94d3-4a12-a4ea-aa262bf53319\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq"
Oct 29 23:53:24.120908 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:24.120903    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc793e2-94d3-4a12-a4ea-aa262bf53319-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq\" (UID: \"4dc793e2-94d3-4a12-a4ea-aa262bf53319\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq"
Oct 29 23:53:24.121642 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:24.121615    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc793e2-94d3-4a12-a4ea-aa262bf53319-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq\" (UID: \"4dc793e2-94d3-4a12-a4ea-aa262bf53319\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq"
Oct 29 23:53:24.123352 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:24.123326    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc793e2-94d3-4a12-a4ea-aa262bf53319-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq\" (UID: \"4dc793e2-94d3-4a12-a4ea-aa262bf53319\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq"
Oct 29 23:53:24.145457 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:24.145432    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn54r\" (UniqueName: \"kubernetes.io/projected/4dc793e2-94d3-4a12-a4ea-aa262bf53319-kube-api-access-rn54r\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq\" (UID: \"4dc793e2-94d3-4a12-a4ea-aa262bf53319\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq"
Oct 29 23:53:24.218155 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:24.218086    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq"
Oct 29 23:53:24.382358 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:24.382327    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:24.525263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:24.525172    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq"]
Oct 29 23:53:24.527972 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:53:24.527930    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc793e2_94d3_4a12_a4ea_aa262bf53319.slice/crio-8c0a8abcaa4ce28389516691b7607b831499d0e023570e155516e2097257d7ef WatchSource:0}: Error finding container 8c0a8abcaa4ce28389516691b7607b831499d0e023570e155516e2097257d7ef: Status 404 returned error can't find the container with id 8c0a8abcaa4ce28389516691b7607b831499d0e023570e155516e2097257d7ef
Oct 29 23:53:24.629768 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:24.629724    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq" event={"ID":"4dc793e2-94d3-4a12-a4ea-aa262bf53319","Type":"ContainerStarted","Data":"8c0a8abcaa4ce28389516691b7607b831499d0e023570e155516e2097257d7ef"}
Oct 29 23:53:24.776826 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:24.776739    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d16a34-4567-4586-a992-297386786fad" path="/var/lib/kubelet/pods/64d16a34-4567-4586-a992-297386786fad/volumes"
Oct 29 23:53:25.274790 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:25.274750    5428 scope.go:122] "RemoveContainer" containerID="65a3081b7e301eadacc7f5de4f0b8136e2b3c84f5be1d8895eea76ec71a8b3c1"
Oct 29 23:53:25.637259 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:25.637222    5428 generic.go:356] "Generic (PLEG): container finished" podID="4dc793e2-94d3-4a12-a4ea-aa262bf53319" containerID="fb36d88dfb3be052d6b705dafb547befb215f0be1ce9aab20e62370d695edc23" exitCode=0
Oct 29 23:53:25.637467 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:25.637311    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq" event={"ID":"4dc793e2-94d3-4a12-a4ea-aa262bf53319","Type":"ContainerDied","Data":"fb36d88dfb3be052d6b705dafb547befb215f0be1ce9aab20e62370d695edc23"}
Oct 29 23:53:25.774182 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:25.774146    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:26.776016 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:26.775990    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq"
Oct 29 23:53:26.853689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:26.853655    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dc793e2-94d3-4a12-a4ea-aa262bf53319-kube-api-access-rn54r\" (UniqueName: \"kubernetes.io/projected/4dc793e2-94d3-4a12-a4ea-aa262bf53319-kube-api-access-rn54r\") pod \"4dc793e2-94d3-4a12-a4ea-aa262bf53319\" (UID: \"4dc793e2-94d3-4a12-a4ea-aa262bf53319\") "
Oct 29 23:53:26.853909 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:26.853755    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dc793e2-94d3-4a12-a4ea-aa262bf53319-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc793e2-94d3-4a12-a4ea-aa262bf53319-maas-api-xks-ca\") pod \"4dc793e2-94d3-4a12-a4ea-aa262bf53319\" (UID: \"4dc793e2-94d3-4a12-a4ea-aa262bf53319\") "
Oct 29 23:53:26.853909 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:26.853815    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dc793e2-94d3-4a12-a4ea-aa262bf53319-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc793e2-94d3-4a12-a4ea-aa262bf53319-maas-api-openshift-ca\") pod \"4dc793e2-94d3-4a12-a4ea-aa262bf53319\" (UID: \"4dc793e2-94d3-4a12-a4ea-aa262bf53319\") "
Oct 29 23:53:26.854160 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:26.854134    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc793e2-94d3-4a12-a4ea-aa262bf53319-maas-api-openshift-ca" pod "4dc793e2-94d3-4a12-a4ea-aa262bf53319" (UID: "4dc793e2-94d3-4a12-a4ea-aa262bf53319"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:53:26.854396 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:26.854374    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dc793e2-94d3-4a12-a4ea-aa262bf53319-maas-api-openshift-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:26.856107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:26.856079    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dc793e2-94d3-4a12-a4ea-aa262bf53319-maas-api-xks-ca" pod "4dc793e2-94d3-4a12-a4ea-aa262bf53319" (UID: "4dc793e2-94d3-4a12-a4ea-aa262bf53319"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:53:26.856221 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:26.856207    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc793e2-94d3-4a12-a4ea-aa262bf53319-kube-api-access-rn54r" pod "4dc793e2-94d3-4a12-a4ea-aa262bf53319" (UID: "4dc793e2-94d3-4a12-a4ea-aa262bf53319"). InnerVolumeSpecName "kube-api-access-rn54r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:53:26.956000 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:26.955927    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn54r\" (UniqueName: \"kubernetes.io/projected/4dc793e2-94d3-4a12-a4ea-aa262bf53319-kube-api-access-rn54r\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:26.956000 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:26.955954    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dc793e2-94d3-4a12-a4ea-aa262bf53319-maas-api-xks-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:27.486529 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:27.486500    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:27.650742 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:27.650705    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq" event={"ID":"4dc793e2-94d3-4a12-a4ea-aa262bf53319","Type":"ContainerDied","Data":"8c0a8abcaa4ce28389516691b7607b831499d0e023570e155516e2097257d7ef"}
Oct 29 23:53:27.650742 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:27.650744    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq"
Oct 29 23:53:27.651058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:27.650757    5428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c0a8abcaa4ce28389516691b7607b831499d0e023570e155516e2097257d7ef"
Oct 29 23:53:29.287723 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:29.287670    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:30.480740 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:30.480703    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:30.594070 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:30.594038    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:30.836000 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:30.835921    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:31.095415 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:31.095331    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:32.419858 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:32.419822    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:32.530176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:32.530145    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:32.955083 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:32.955047    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:33.714968 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:33.714935    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:33.830559 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:33.830524    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:34.147830 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:34.147773    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:34.300349 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:34.300314    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45"]
Oct 29 23:53:34.331413 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:34.331382    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd"]
Oct 29 23:53:34.441699 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:34.441616    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:34.568668 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:34.568635    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:34.921203 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:34.921167    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:35.145653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:35.145623    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:36.641563 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:36.641536    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:37.333585 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:37.333550    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:37.450925 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:37.450891    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:37.485000 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:37.484967    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq"]
Oct 29 23:53:37.496503 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:37.496476    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-383f0e-pvczq"]
Oct 29 23:53:37.561545 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:37.561516    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:37.971406 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:37.971372    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:38.776344 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:38.776305    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dc793e2-94d3-4a12-a4ea-aa262bf53319" path="/var/lib/kubelet/pods/4dc793e2-94d3-4a12-a4ea-aa262bf53319/volumes"
Oct 29 23:53:39.301547 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.301504    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45" podUID="b46913cc-66af-454a-8d7f-0cff4a4b9d76" containerName="payload-processing" containerID="cri-o://ad0013336aa7bc0dfe7dcd8c8ddb4824b4182e255024216a5452470e21e90eae" gracePeriod=25
Oct 29 23:53:39.332504 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.332475    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd" podUID="1f34098a-acd0-4c75-b946-0de1616e0ba8" containerName="payload-pre-processing" containerID="cri-o://40c015e19e10c402eb3e49c8601aa5817f2a1bf23f72cbe52a645d26331ab21b" gracePeriod=25
Oct 29 23:53:39.482322 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.482278    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:39.607284 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.607258    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45"
Oct 29 23:53:39.660035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.660007    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd"
Oct 29 23:53:39.730225 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.730189    5428 generic.go:356] "Generic (PLEG): container finished" podID="b46913cc-66af-454a-8d7f-0cff4a4b9d76" containerID="ad0013336aa7bc0dfe7dcd8c8ddb4824b4182e255024216a5452470e21e90eae" exitCode=0
Oct 29 23:53:39.730428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.730253    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45"
Oct 29 23:53:39.730428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.730267    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45" event={"ID":"b46913cc-66af-454a-8d7f-0cff4a4b9d76","Type":"ContainerDied","Data":"ad0013336aa7bc0dfe7dcd8c8ddb4824b4182e255024216a5452470e21e90eae"}
Oct 29 23:53:39.730428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.730321    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45" event={"ID":"b46913cc-66af-454a-8d7f-0cff4a4b9d76","Type":"ContainerDied","Data":"15b47c7adaf57036187178cef3da4d7d81c0b7115ac7e7672e360a678c31a4ad"}
Oct 29 23:53:39.730428 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.730349    5428 scope.go:122] "RemoveContainer" containerID="ad0013336aa7bc0dfe7dcd8c8ddb4824b4182e255024216a5452470e21e90eae"
Oct 29 23:53:39.731794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.731769    5428 generic.go:356] "Generic (PLEG): container finished" podID="1f34098a-acd0-4c75-b946-0de1616e0ba8" containerID="40c015e19e10c402eb3e49c8601aa5817f2a1bf23f72cbe52a645d26331ab21b" exitCode=0
Oct 29 23:53:39.731936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.731841    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd"
Oct 29 23:53:39.731936 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.731922    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd" event={"ID":"1f34098a-acd0-4c75-b946-0de1616e0ba8","Type":"ContainerDied","Data":"40c015e19e10c402eb3e49c8601aa5817f2a1bf23f72cbe52a645d26331ab21b"}
Oct 29 23:53:39.732053 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.731955    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd" event={"ID":"1f34098a-acd0-4c75-b946-0de1616e0ba8","Type":"ContainerDied","Data":"0e9190b92f117abef6879e17d0994eeb4b598cc9e255b3220874b00ece5d916c"}
Oct 29 23:53:39.741614 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.741588    5428 scope.go:122] "RemoveContainer" containerID="ad0013336aa7bc0dfe7dcd8c8ddb4824b4182e255024216a5452470e21e90eae"
Oct 29 23:53:39.741973 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:53:39.741952    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad0013336aa7bc0dfe7dcd8c8ddb4824b4182e255024216a5452470e21e90eae\": container with ID starting with ad0013336aa7bc0dfe7dcd8c8ddb4824b4182e255024216a5452470e21e90eae not found: ID does not exist" containerID="ad0013336aa7bc0dfe7dcd8c8ddb4824b4182e255024216a5452470e21e90eae"
Oct 29 23:53:39.742048 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.741980    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad0013336aa7bc0dfe7dcd8c8ddb4824b4182e255024216a5452470e21e90eae"} err="failed to get container status \"ad0013336aa7bc0dfe7dcd8c8ddb4824b4182e255024216a5452470e21e90eae\": rpc error: code = NotFound desc = could not find container \"ad0013336aa7bc0dfe7dcd8c8ddb4824b4182e255024216a5452470e21e90eae\": container with ID starting with ad0013336aa7bc0dfe7dcd8c8ddb4824b4182e255024216a5452470e21e90eae not found: ID does not exist"
Oct 29 23:53:39.742048 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.741998    5428 scope.go:122] "RemoveContainer" containerID="40c015e19e10c402eb3e49c8601aa5817f2a1bf23f72cbe52a645d26331ab21b"
Oct 29 23:53:39.754263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.754244    5428 scope.go:122] "RemoveContainer" containerID="40c015e19e10c402eb3e49c8601aa5817f2a1bf23f72cbe52a645d26331ab21b"
Oct 29 23:53:39.754595 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:53:39.754572    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c015e19e10c402eb3e49c8601aa5817f2a1bf23f72cbe52a645d26331ab21b\": container with ID starting with 40c015e19e10c402eb3e49c8601aa5817f2a1bf23f72cbe52a645d26331ab21b not found: ID does not exist" containerID="40c015e19e10c402eb3e49c8601aa5817f2a1bf23f72cbe52a645d26331ab21b"
Oct 29 23:53:39.754688 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.754604    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c015e19e10c402eb3e49c8601aa5817f2a1bf23f72cbe52a645d26331ab21b"} err="failed to get container status \"40c015e19e10c402eb3e49c8601aa5817f2a1bf23f72cbe52a645d26331ab21b\": rpc error: code = NotFound desc = could not find container \"40c015e19e10c402eb3e49c8601aa5817f2a1bf23f72cbe52a645d26331ab21b\": container with ID starting with 40c015e19e10c402eb3e49c8601aa5817f2a1bf23f72cbe52a645d26331ab21b not found: ID does not exist"
Oct 29 23:53:39.799060 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.799020    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46913cc-66af-454a-8d7f-0cff4a4b9d76-kube-api-access-8twr2\" (UniqueName: \"kubernetes.io/projected/b46913cc-66af-454a-8d7f-0cff4a4b9d76-kube-api-access-8twr2\") pod \"b46913cc-66af-454a-8d7f-0cff4a4b9d76\" (UID: \"b46913cc-66af-454a-8d7f-0cff4a4b9d76\") "
Oct 29 23:53:39.799237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.799069    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46913cc-66af-454a-8d7f-0cff4a4b9d76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b46913cc-66af-454a-8d7f-0cff4a4b9d76-plugins-config-volume\") pod \"b46913cc-66af-454a-8d7f-0cff4a4b9d76\" (UID: \"b46913cc-66af-454a-8d7f-0cff4a4b9d76\") "
Oct 29 23:53:39.799237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.799104    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f34098a-acd0-4c75-b946-0de1616e0ba8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f34098a-acd0-4c75-b946-0de1616e0ba8-plugins-config-volume\") pod \"1f34098a-acd0-4c75-b946-0de1616e0ba8\" (UID: \"1f34098a-acd0-4c75-b946-0de1616e0ba8\") "
Oct 29 23:53:39.799237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.799148    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f34098a-acd0-4c75-b946-0de1616e0ba8-kube-api-access-z8q8v\" (UniqueName: \"kubernetes.io/projected/1f34098a-acd0-4c75-b946-0de1616e0ba8-kube-api-access-z8q8v\") pod \"1f34098a-acd0-4c75-b946-0de1616e0ba8\" (UID: \"1f34098a-acd0-4c75-b946-0de1616e0ba8\") "
Oct 29 23:53:39.799564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.799539    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f34098a-acd0-4c75-b946-0de1616e0ba8-plugins-config-volume" pod "1f34098a-acd0-4c75-b946-0de1616e0ba8" (UID: "1f34098a-acd0-4c75-b946-0de1616e0ba8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:53:39.799653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.799560    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46913cc-66af-454a-8d7f-0cff4a4b9d76-plugins-config-volume" pod "b46913cc-66af-454a-8d7f-0cff4a4b9d76" (UID: "b46913cc-66af-454a-8d7f-0cff4a4b9d76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:53:39.801388 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.801364    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f34098a-acd0-4c75-b946-0de1616e0ba8-kube-api-access-z8q8v" pod "1f34098a-acd0-4c75-b946-0de1616e0ba8" (UID: "1f34098a-acd0-4c75-b946-0de1616e0ba8"). InnerVolumeSpecName "kube-api-access-z8q8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:53:39.801472 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.801406    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46913cc-66af-454a-8d7f-0cff4a4b9d76-kube-api-access-8twr2" pod "b46913cc-66af-454a-8d7f-0cff4a4b9d76" (UID: "b46913cc-66af-454a-8d7f-0cff4a4b9d76"). InnerVolumeSpecName "kube-api-access-8twr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:53:39.900281 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.900252    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8twr2\" (UniqueName: \"kubernetes.io/projected/b46913cc-66af-454a-8d7f-0cff4a4b9d76-kube-api-access-8twr2\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:39.900281 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.900275    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b46913cc-66af-454a-8d7f-0cff4a4b9d76-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:39.900281 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.900285    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f34098a-acd0-4c75-b946-0de1616e0ba8-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:39.900525 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:39.900295    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8q8v\" (UniqueName: \"kubernetes.io/projected/1f34098a-acd0-4c75-b946-0de1616e0ba8-kube-api-access-z8q8v\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:40.079967 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:40.079934    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd"]
Oct 29 23:53:40.091875 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:40.091838    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-383f0e-7954f5ccd7-dc4qd"]
Oct 29 23:53:40.118032 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:40.118004    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45"]
Oct 29 23:53:40.122556 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:40.122527    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-383f0e-7cdfc54bf8-gbz45"]
Oct 29 23:53:40.775894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:40.775865    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f34098a-acd0-4c75-b946-0de1616e0ba8" path="/var/lib/kubelet/pods/1f34098a-acd0-4c75-b946-0de1616e0ba8/volumes"
Oct 29 23:53:40.776281 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:40.776255    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46913cc-66af-454a-8d7f-0cff4a4b9d76" path="/var/lib/kubelet/pods/b46913cc-66af-454a-8d7f-0cff4a4b9d76/volumes"
Oct 29 23:53:41.298267 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:41.298230    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:41.457523 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:41.457485    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"]
Oct 29 23:53:41.457735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:41.457716    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc" podUID="fe45dacc-3544-4123-a779-db0455a8a702" containerName="istio-proxy" containerID="cri-o://d4c86113710102dde56aeceec7a58779c511bcca03b39cf5cdaff2a936a43506" gracePeriod=30
Oct 29 23:53:41.969698 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:41.969669    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:42.088278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:42.088245    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:43.090413 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:43.090375    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:44.196630 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:44.196595    5428 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:53:44.197017 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:44.196651    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc" podUID="fe45dacc-3544-4123-a779-db0455a8a702" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:53:44.692961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:44.692916    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:44.802080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:44.802038    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:45.037712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:45.037625    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:45.318325 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:45.318210    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:45.438566 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:45.438532    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:45.560733 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:45.560697    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:45.999154 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:45.999122    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:46.725538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.725482    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:53:46.774589 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.774554    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-workload-socket\") pod \"fe45dacc-3544-4123-a779-db0455a8a702\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") "
Oct 29 23:53:46.774589 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.774592    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-istio-envoy\") pod \"fe45dacc-3544-4123-a779-db0455a8a702\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") "
Oct 29 23:53:46.774848 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.774650    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe45dacc-3544-4123-a779-db0455a8a702-istio-token\" (UniqueName: \"kubernetes.io/projected/fe45dacc-3544-4123-a779-db0455a8a702-istio-token\") pod \"fe45dacc-3544-4123-a779-db0455a8a702\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") "
Oct 29 23:53:46.774848 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.774751    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-credential-socket\") pod \"fe45dacc-3544-4123-a779-db0455a8a702\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") "
Oct 29 23:53:46.774848 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.774795    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe45dacc-3544-4123-a779-db0455a8a702-kube-api-access-w7jxs\" (UniqueName: \"kubernetes.io/projected/fe45dacc-3544-4123-a779-db0455a8a702-kube-api-access-w7jxs\") pod \"fe45dacc-3544-4123-a779-db0455a8a702\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") "
Oct 29 23:53:46.775029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.774877    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe45dacc-3544-4123-a779-db0455a8a702-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe45dacc-3544-4123-a779-db0455a8a702-istio-podinfo\") pod \"fe45dacc-3544-4123-a779-db0455a8a702\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") "
Oct 29 23:53:46.775029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.774937    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe45dacc-3544-4123-a779-db0455a8a702-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe45dacc-3544-4123-a779-db0455a8a702-istiod-ca-cert\") pod \"fe45dacc-3544-4123-a779-db0455a8a702\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") "
Oct 29 23:53:46.775029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.774969    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-istio-data\") pod \"fe45dacc-3544-4123-a779-db0455a8a702\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") "
Oct 29 23:53:46.775029 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.774872    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-workload-socket" pod "fe45dacc-3544-4123-a779-db0455a8a702" (UID: "fe45dacc-3544-4123-a779-db0455a8a702"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:53:46.775235 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.775051    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-workload-certs\") pod \"fe45dacc-3544-4123-a779-db0455a8a702\" (UID: \"fe45dacc-3544-4123-a779-db0455a8a702\") "
Oct 29 23:53:46.775351 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.775323    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-credential-socket" pod "fe45dacc-3544-4123-a779-db0455a8a702" (UID: "fe45dacc-3544-4123-a779-db0455a8a702"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:53:46.775429 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.775368    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe45dacc-3544-4123-a779-db0455a8a702-istiod-ca-cert" pod "fe45dacc-3544-4123-a779-db0455a8a702" (UID: "fe45dacc-3544-4123-a779-db0455a8a702"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:53:46.775492 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.775474    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:46.775555 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.775493    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe45dacc-3544-4123-a779-db0455a8a702-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:46.775555 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.775508    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:46.775665 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.775621    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-istio-data" pod "fe45dacc-3544-4123-a779-db0455a8a702" (UID: "fe45dacc-3544-4123-a779-db0455a8a702"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:53:46.775926 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.775904    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-workload-certs" pod "fe45dacc-3544-4123-a779-db0455a8a702" (UID: "fe45dacc-3544-4123-a779-db0455a8a702"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:53:46.777624 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.777599    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe45dacc-3544-4123-a779-db0455a8a702-istio-podinfo" pod "fe45dacc-3544-4123-a779-db0455a8a702" (UID: "fe45dacc-3544-4123-a779-db0455a8a702"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:53:46.777900 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.777874    5428 generic.go:356] "Generic (PLEG): container finished" podID="fe45dacc-3544-4123-a779-db0455a8a702" containerID="d4c86113710102dde56aeceec7a58779c511bcca03b39cf5cdaff2a936a43506" exitCode=0
Oct 29 23:53:46.778013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.777995    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"
Oct 29 23:53:46.778137 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.778114    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe45dacc-3544-4123-a779-db0455a8a702-kube-api-access-w7jxs" pod "fe45dacc-3544-4123-a779-db0455a8a702" (UID: "fe45dacc-3544-4123-a779-db0455a8a702"). InnerVolumeSpecName "kube-api-access-w7jxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:53:46.778211 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.778146    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-istio-envoy" pod "fe45dacc-3544-4123-a779-db0455a8a702" (UID: "fe45dacc-3544-4123-a779-db0455a8a702"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:53:46.779257 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.779241    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe45dacc-3544-4123-a779-db0455a8a702-istio-token" pod "fe45dacc-3544-4123-a779-db0455a8a702" (UID: "fe45dacc-3544-4123-a779-db0455a8a702"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:53:46.798407 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.798381    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc" event={"ID":"fe45dacc-3544-4123-a779-db0455a8a702","Type":"ContainerDied","Data":"d4c86113710102dde56aeceec7a58779c511bcca03b39cf5cdaff2a936a43506"}
Oct 29 23:53:46.798548 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.798435    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc" event={"ID":"fe45dacc-3544-4123-a779-db0455a8a702","Type":"ContainerDied","Data":"2dd9140b4d9743feab861e2924e768c24a7f221e59776c83b4df21dee53764ef"}
Oct 29 23:53:46.798548 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.798464    5428 scope.go:122] "RemoveContainer" containerID="d4c86113710102dde56aeceec7a58779c511bcca03b39cf5cdaff2a936a43506"
Oct 29 23:53:46.809098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.809073    5428 scope.go:122] "RemoveContainer" containerID="d4c86113710102dde56aeceec7a58779c511bcca03b39cf5cdaff2a936a43506"
Oct 29 23:53:46.809429 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:53:46.809405    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4c86113710102dde56aeceec7a58779c511bcca03b39cf5cdaff2a936a43506\": container with ID starting with d4c86113710102dde56aeceec7a58779c511bcca03b39cf5cdaff2a936a43506 not found: ID does not exist" containerID="d4c86113710102dde56aeceec7a58779c511bcca03b39cf5cdaff2a936a43506"
Oct 29 23:53:46.809513 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.809442    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4c86113710102dde56aeceec7a58779c511bcca03b39cf5cdaff2a936a43506"} err="failed to get container status \"d4c86113710102dde56aeceec7a58779c511bcca03b39cf5cdaff2a936a43506\": rpc error: code = NotFound desc = could not find container \"d4c86113710102dde56aeceec7a58779c511bcca03b39cf5cdaff2a936a43506\": container with ID starting with d4c86113710102dde56aeceec7a58779c511bcca03b39cf5cdaff2a936a43506 not found: ID does not exist"
Oct 29 23:53:46.876485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.876455    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:46.876485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.876486    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:46.876748 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.876499    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe45dacc-3544-4123-a779-db0455a8a702-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:46.876748 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.876513    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7jxs\" (UniqueName: \"kubernetes.io/projected/fe45dacc-3544-4123-a779-db0455a8a702-kube-api-access-w7jxs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:46.876748 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.876526    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe45dacc-3544-4123-a779-db0455a8a702-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:46.876748 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:46.876539    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe45dacc-3544-4123-a779-db0455a8a702-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:47.123626 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:47.123593    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"]
Oct 29 23:53:47.135817 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:47.135773    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2126fb-openshift-default-5c7df488f5-wk6fc"]
Oct 29 23:53:47.367901 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:53:47.367862    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:53:47.522126 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:47.522097    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:48.775930 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:48.775898    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe45dacc-3544-4123-a779-db0455a8a702" path="/var/lib/kubelet/pods/fe45dacc-3544-4123-a779-db0455a8a702/volumes"
Oct 29 23:53:49.290153 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:49.290119    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:50.592510 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:50.592476    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:51.057192 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:51.057159    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:51.749863 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:51.749832    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:51.869506 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:51.869468    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:52.084041 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:52.083957    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:52.789965 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:52.789934    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:54.254474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.254435    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"]
Oct 29 23:53:54.255327 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.254884    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz" podUID="9e1067f9-ce81-450a-9a8b-8603f7866708" containerName="maas-api" containerID="cri-o://63626851e753e6b0ecf631602800d5c284ecfafb72c133e596b0a79b56126e42" gracePeriod=30
Oct 29 23:53:54.334916 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.334885    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:54.523055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.523029    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"
Oct 29 23:53:54.657927 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.657893    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e1067f9-ce81-450a-9a8b-8603f7866708-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1067f9-ce81-450a-9a8b-8603f7866708-openshift-service-ca\") pod \"9e1067f9-ce81-450a-9a8b-8603f7866708\" (UID: \"9e1067f9-ce81-450a-9a8b-8603f7866708\") "
Oct 29 23:53:54.658111 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.657945    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e1067f9-ce81-450a-9a8b-8603f7866708-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e1067f9-ce81-450a-9a8b-8603f7866708-metrics-certs\") pod \"9e1067f9-ce81-450a-9a8b-8603f7866708\" (UID: \"9e1067f9-ce81-450a-9a8b-8603f7866708\") "
Oct 29 23:53:54.658111 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.657973    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1067f9-ce81-450a-9a8b-8603f7866708-kube-api-access-htzs5\" (UniqueName: \"kubernetes.io/projected/9e1067f9-ce81-450a-9a8b-8603f7866708-kube-api-access-htzs5\") pod \"9e1067f9-ce81-450a-9a8b-8603f7866708\" (UID: \"9e1067f9-ce81-450a-9a8b-8603f7866708\") "
Oct 29 23:53:54.658111 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.658021    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e1067f9-ce81-450a-9a8b-8603f7866708-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1067f9-ce81-450a-9a8b-8603f7866708-maas-api-tls\") pod \"9e1067f9-ce81-450a-9a8b-8603f7866708\" (UID: \"9e1067f9-ce81-450a-9a8b-8603f7866708\") "
Oct 29 23:53:54.658322 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.658301    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1067f9-ce81-450a-9a8b-8603f7866708-openshift-service-ca" pod "9e1067f9-ce81-450a-9a8b-8603f7866708" (UID: "9e1067f9-ce81-450a-9a8b-8603f7866708"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:53:54.658465 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.658444    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1067f9-ce81-450a-9a8b-8603f7866708-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:54.660106 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.660080    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e1067f9-ce81-450a-9a8b-8603f7866708-maas-api-tls" pod "9e1067f9-ce81-450a-9a8b-8603f7866708" (UID: "9e1067f9-ce81-450a-9a8b-8603f7866708"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:53:54.660106 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.660088    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e1067f9-ce81-450a-9a8b-8603f7866708-metrics-certs" pod "9e1067f9-ce81-450a-9a8b-8603f7866708" (UID: "9e1067f9-ce81-450a-9a8b-8603f7866708"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:53:54.660252 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.660177    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1067f9-ce81-450a-9a8b-8603f7866708-kube-api-access-htzs5" pod "9e1067f9-ce81-450a-9a8b-8603f7866708" (UID: "9e1067f9-ce81-450a-9a8b-8603f7866708"). InnerVolumeSpecName "kube-api-access-htzs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:53:54.759684 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.759605    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e1067f9-ce81-450a-9a8b-8603f7866708-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:54.759684 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.759633    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htzs5\" (UniqueName: \"kubernetes.io/projected/9e1067f9-ce81-450a-9a8b-8603f7866708-kube-api-access-htzs5\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:54.759684 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.759645    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1067f9-ce81-450a-9a8b-8603f7866708-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:53:54.824998 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.824960    5428 generic.go:356] "Generic (PLEG): container finished" podID="9e1067f9-ce81-450a-9a8b-8603f7866708" containerID="63626851e753e6b0ecf631602800d5c284ecfafb72c133e596b0a79b56126e42" exitCode=0
Oct 29 23:53:54.825165 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.825008    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"
Oct 29 23:53:54.825165 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.825041    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz" event={"ID":"9e1067f9-ce81-450a-9a8b-8603f7866708","Type":"ContainerDied","Data":"63626851e753e6b0ecf631602800d5c284ecfafb72c133e596b0a79b56126e42"}
Oct 29 23:53:54.825165 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.825098    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz" event={"ID":"9e1067f9-ce81-450a-9a8b-8603f7866708","Type":"ContainerDied","Data":"02afc7204ba925eeccaf689a1b6a4c3ab640f0fe806d616d1ef44a7bb11f6418"}
Oct 29 23:53:54.825165 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.825123    5428 scope.go:122] "RemoveContainer" containerID="63626851e753e6b0ecf631602800d5c284ecfafb72c133e596b0a79b56126e42"
Oct 29 23:53:54.835389 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.835351    5428 scope.go:122] "RemoveContainer" containerID="63626851e753e6b0ecf631602800d5c284ecfafb72c133e596b0a79b56126e42"
Oct 29 23:53:54.835674 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:53:54.835654    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63626851e753e6b0ecf631602800d5c284ecfafb72c133e596b0a79b56126e42\": container with ID starting with 63626851e753e6b0ecf631602800d5c284ecfafb72c133e596b0a79b56126e42 not found: ID does not exist" containerID="63626851e753e6b0ecf631602800d5c284ecfafb72c133e596b0a79b56126e42"
Oct 29 23:53:54.835762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.835682    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63626851e753e6b0ecf631602800d5c284ecfafb72c133e596b0a79b56126e42"} err="failed to get container status \"63626851e753e6b0ecf631602800d5c284ecfafb72c133e596b0a79b56126e42\": rpc error: code = NotFound desc = could not find container \"63626851e753e6b0ecf631602800d5c284ecfafb72c133e596b0a79b56126e42\": container with ID starting with 63626851e753e6b0ecf631602800d5c284ecfafb72c133e596b0a79b56126e42 not found: ID does not exist"
Oct 29 23:53:54.859850 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.859826    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:54.878469 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.878442    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"]
Oct 29 23:53:54.890377 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.890347    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e2ac7-6c99dc7cd5-kr6qz"]
Oct 29 23:53:54.979985 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:54.979956    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:55.109264 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:55.109161    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:56.776024 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:56.775994    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e1067f9-ce81-450a-9a8b-8603f7866708" path="/var/lib/kubelet/pods/9e1067f9-ce81-450a-9a8b-8603f7866708/volumes"
Oct 29 23:53:56.938562 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:56.938531    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:53:58.790929 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:53:58.790859    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:00.490951 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:00.490918    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:02.184742 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:02.184710    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:04.078065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:04.078026    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:05.497383 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:05.497348    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:05.892282 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:05.892195    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:06.304440 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:06.304322    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:06.419783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:06.419748    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:07.693078 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:07.693038    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:09.589545 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:09.589513    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:11.398710 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:11.398678    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:11.703216 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:11.703129    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:11.819510 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:11.819463    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:12.034746 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:12.034665    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:12.536687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:12.536656    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:13.393946 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:13.393912    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp"]
Oct 29 23:54:13.395083 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:13.395046    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:13.546336 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:13.546306    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:13.657098 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:13.657013    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:15.274031 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:15.273998    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:16.785265 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.785226    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75"]
Oct 29 23:54:16.785929 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.785893    5428 cpu_manager.go:414] "containerMap: removing container" podUID="1f34098a-acd0-4c75-b946-0de1616e0ba8" containerName="payload-pre-processing"
Oct 29 23:54:16.785929 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.785921    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f34098a-acd0-4c75-b946-0de1616e0ba8" containerName="payload-pre-processing"
Oct 29 23:54:16.786130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.785938    5428 cpu_manager.go:414] "containerMap: removing container" podUID="fe45dacc-3544-4123-a779-db0455a8a702" containerName="istio-proxy"
Oct 29 23:54:16.786130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.785947    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe45dacc-3544-4123-a779-db0455a8a702" containerName="istio-proxy"
Oct 29 23:54:16.786130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.785977    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9e1067f9-ce81-450a-9a8b-8603f7866708" containerName="maas-api"
Oct 29 23:54:16.786130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.785985    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1067f9-ce81-450a-9a8b-8603f7866708" containerName="maas-api"
Oct 29 23:54:16.786130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.786009    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b46913cc-66af-454a-8d7f-0cff4a4b9d76" containerName="payload-processing"
Oct 29 23:54:16.786130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.786017    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46913cc-66af-454a-8d7f-0cff4a4b9d76" containerName="payload-processing"
Oct 29 23:54:16.786130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.786043    5428 cpu_manager.go:414] "containerMap: removing container" podUID="4dc793e2-94d3-4a12-a4ea-aa262bf53319" containerName="revoke-keys"
Oct 29 23:54:16.786130 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.786050    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dc793e2-94d3-4a12-a4ea-aa262bf53319" containerName="revoke-keys"
Oct 29 23:54:16.786535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.786139    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f34098a-acd0-4c75-b946-0de1616e0ba8" containerName="payload-pre-processing"
Oct 29 23:54:16.786535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.786153    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe45dacc-3544-4123-a779-db0455a8a702" containerName="istio-proxy"
Oct 29 23:54:16.786535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.786165    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1067f9-ce81-450a-9a8b-8603f7866708" containerName="maas-api"
Oct 29 23:54:16.786535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.786176    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46913cc-66af-454a-8d7f-0cff4a4b9d76" containerName="payload-processing"
Oct 29 23:54:16.786535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.786188    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dc793e2-94d3-4a12-a4ea-aa262bf53319" containerName="revoke-keys"
Oct 29 23:54:16.790118 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.790093    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75"
Oct 29 23:54:16.795244 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.795212    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqtx4\" (UniqueName: \"kubernetes.io/projected/84aa240d-42ed-43c5-a84b-001a707311ba-kube-api-access-lqtx4\") pod \"maas-api-revoke-sub-57bb2a42-8pf75\" (UID: \"84aa240d-42ed-43c5-a84b-001a707311ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75"
Oct 29 23:54:16.795480 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.795449    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aa240d-42ed-43c5-a84b-001a707311ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57bb2a42-8pf75\" (UID: \"84aa240d-42ed-43c5-a84b-001a707311ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75"
Oct 29 23:54:16.795612 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.795506    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aa240d-42ed-43c5-a84b-001a707311ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57bb2a42-8pf75\" (UID: \"84aa240d-42ed-43c5-a84b-001a707311ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75"
Oct 29 23:54:16.807739 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.807702    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75"]
Oct 29 23:54:16.885437 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.885403    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:54:16.885653 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.885634    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\""
Oct 29 23:54:16.896314 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.896287    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aa240d-42ed-43c5-a84b-001a707311ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57bb2a42-8pf75\" (UID: \"84aa240d-42ed-43c5-a84b-001a707311ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75"
Oct 29 23:54:16.896426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.896334    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aa240d-42ed-43c5-a84b-001a707311ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57bb2a42-8pf75\" (UID: \"84aa240d-42ed-43c5-a84b-001a707311ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75"
Oct 29 23:54:16.896426 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.896374    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqtx4\" (UniqueName: \"kubernetes.io/projected/84aa240d-42ed-43c5-a84b-001a707311ba-kube-api-access-lqtx4\") pod \"maas-api-revoke-sub-57bb2a42-8pf75\" (UID: \"84aa240d-42ed-43c5-a84b-001a707311ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75"
Oct 29 23:54:16.897055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.897031    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aa240d-42ed-43c5-a84b-001a707311ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57bb2a42-8pf75\" (UID: \"84aa240d-42ed-43c5-a84b-001a707311ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75"
Oct 29 23:54:16.898617 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.898602    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aa240d-42ed-43c5-a84b-001a707311ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57bb2a42-8pf75\" (UID: \"84aa240d-42ed-43c5-a84b-001a707311ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75"
Oct 29 23:54:16.919439 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:16.919413    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqtx4\" (UniqueName: \"kubernetes.io/projected/84aa240d-42ed-43c5-a84b-001a707311ba-kube-api-access-lqtx4\") pod \"maas-api-revoke-sub-57bb2a42-8pf75\" (UID: \"84aa240d-42ed-43c5-a84b-001a707311ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75"
Oct 29 23:54:17.105532 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:17.105446    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75"
Oct 29 23:54:17.193766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:17.193740    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:17.369860 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:54:17.369795    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:54:17.412353 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:17.412297    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75"]
Oct 29 23:54:17.415050 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:54:17.415020    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84aa240d_42ed_43c5_a84b_001a707311ba.slice/crio-8cbd06b1b7718710aab372ee45f702075a5657ad4b3f8239861aab7a25ace029 WatchSource:0}: Error finding container 8cbd06b1b7718710aab372ee45f702075a5657ad4b3f8239861aab7a25ace029: Status 404 returned error can't find the container with id 8cbd06b1b7718710aab372ee45f702075a5657ad4b3f8239861aab7a25ace029
Oct 29 23:54:17.501290 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:17.501265    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"]
Oct 29 23:54:17.509564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:17.509540    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"]
Oct 29 23:54:17.624784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:17.624707    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:17.958674 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:17.958592    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75" event={"ID":"84aa240d-42ed-43c5-a84b-001a707311ba","Type":"ContainerStarted","Data":"abc3ca3f192bf7481fcf10318db5a4c314fb324824614b523ea42c91abc5cc52"}
Oct 29 23:54:17.958674 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:17.958645    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75" event={"ID":"84aa240d-42ed-43c5-a84b-001a707311ba","Type":"ContainerStarted","Data":"8cbd06b1b7718710aab372ee45f702075a5657ad4b3f8239861aab7a25ace029"}
Oct 29 23:54:17.999221 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:17.999173    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75" podStartSLOduration=1.999158919 podStartE2EDuration="1.999158919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:54:17.997990774 +0000 UTC m=+4261.340939520" watchObservedRunningTime="2026-10-29 23:54:17.999158919 +0000 UTC m=+4261.342107664"
Oct 29 23:54:18.394697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:18.394660    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp" podUID="b73ad056-0a89-4c23-84b7-d0eceab57c0b" containerName="payload-processing" containerID="cri-o://a55be33c86914b092d49c10b843ab03c9e2ff7abe48cb092c1b69dbdd10eb2ae" gracePeriod=25
Oct 29 23:54:18.965624 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:18.965589    5428 generic.go:356] "Generic (PLEG): container finished" podID="84aa240d-42ed-43c5-a84b-001a707311ba" containerID="abc3ca3f192bf7481fcf10318db5a4c314fb324824614b523ea42c91abc5cc52" exitCode=0
Oct 29 23:54:18.966071 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:18.965675    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75" event={"ID":"84aa240d-42ed-43c5-a84b-001a707311ba","Type":"ContainerDied","Data":"abc3ca3f192bf7481fcf10318db5a4c314fb324824614b523ea42c91abc5cc52"}
Oct 29 23:54:18.966703 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:18.966685    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:20.118988 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:20.118959    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75"
Oct 29 23:54:20.230233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:20.230147    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84aa240d-42ed-43c5-a84b-001a707311ba-kube-api-access-lqtx4\" (UniqueName: \"kubernetes.io/projected/84aa240d-42ed-43c5-a84b-001a707311ba-kube-api-access-lqtx4\") pod \"84aa240d-42ed-43c5-a84b-001a707311ba\" (UID: \"84aa240d-42ed-43c5-a84b-001a707311ba\") "
Oct 29 23:54:20.230233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:20.230202    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84aa240d-42ed-43c5-a84b-001a707311ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aa240d-42ed-43c5-a84b-001a707311ba-maas-api-openshift-ca\") pod \"84aa240d-42ed-43c5-a84b-001a707311ba\" (UID: \"84aa240d-42ed-43c5-a84b-001a707311ba\") "
Oct 29 23:54:20.230435 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:20.230313    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84aa240d-42ed-43c5-a84b-001a707311ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aa240d-42ed-43c5-a84b-001a707311ba-maas-api-xks-ca\") pod \"84aa240d-42ed-43c5-a84b-001a707311ba\" (UID: \"84aa240d-42ed-43c5-a84b-001a707311ba\") "
Oct 29 23:54:20.230550 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:20.230525    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84aa240d-42ed-43c5-a84b-001a707311ba-maas-api-openshift-ca" pod "84aa240d-42ed-43c5-a84b-001a707311ba" (UID: "84aa240d-42ed-43c5-a84b-001a707311ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:54:20.230762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:20.230743    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aa240d-42ed-43c5-a84b-001a707311ba-maas-api-openshift-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:20.232366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:20.232343    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84aa240d-42ed-43c5-a84b-001a707311ba-maas-api-xks-ca" pod "84aa240d-42ed-43c5-a84b-001a707311ba" (UID: "84aa240d-42ed-43c5-a84b-001a707311ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:54:20.232468 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:20.232343    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84aa240d-42ed-43c5-a84b-001a707311ba-kube-api-access-lqtx4" pod "84aa240d-42ed-43c5-a84b-001a707311ba" (UID: "84aa240d-42ed-43c5-a84b-001a707311ba"). InnerVolumeSpecName "kube-api-access-lqtx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:54:20.331780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:20.331752    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqtx4\" (UniqueName: \"kubernetes.io/projected/84aa240d-42ed-43c5-a84b-001a707311ba-kube-api-access-lqtx4\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:20.331780 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:20.331778    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aa240d-42ed-43c5-a84b-001a707311ba-maas-api-xks-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:20.417867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:20.417811    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:20.970723 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:20.970672    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:20.977515 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:20.977489    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75"
Oct 29 23:54:20.977638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:20.977519    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bb2a42-8pf75" event={"ID":"84aa240d-42ed-43c5-a84b-001a707311ba","Type":"ContainerDied","Data":"8cbd06b1b7718710aab372ee45f702075a5657ad4b3f8239861aab7a25ace029"}
Oct 29 23:54:20.977638 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:20.977568    5428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cbd06b1b7718710aab372ee45f702075a5657ad4b3f8239861aab7a25ace029"
Oct 29 23:54:21.094728 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:21.094647    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:21.629996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:21.629963    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:23.033067 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:23.033035    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:23.443701 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:23.443666    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:23.560660 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:23.560624    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:24.881333 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:24.881295    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:24.978554 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:24.978513    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp" podUID="b73ad056-0a89-4c23-84b7-d0eceab57c0b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:54:26.037444 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:26.037406    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:26.148340 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:26.148308    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:26.811181 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:26.811148    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:28.393707 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:28.393672    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:29.015320 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:29.015285    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:29.140209 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:29.140176    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:29.251961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:29.251915    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:29.457566 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:29.457525    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:29.650920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:29.650885    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:29.782156 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:29.782073    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:29.949908 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:29.949871    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:30.163485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:30.163449    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:30.492354 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:30.492275    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:30.694398 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:30.694366    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:30.807278 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:30.807195    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:31.048251 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:31.048223    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:31.191152 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:31.191100    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:32.522145 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.522112    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" containerID="cri-o://dc51cb2a3dfcb15cd3f362c831edb227ba0ee579bc12a8ba57f2386b12211073" gracePeriod=45
Oct 29 23:54:32.530077 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.530046    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main" containerID="cri-o://61dde9e7557330a23f0882a0c7b222aec7cf5a8538d053a85aa64435afc892e2" gracePeriod=45
Oct 29 23:54:32.809349 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.809322    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 23:54:32.834970 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.834946    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 23:54:32.872493 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.872456    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-model-cache\") pod \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") "
Oct 29 23:54:32.872883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.872821    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-model-cache" pod "b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" (UID: "b8ada4d0-3756-4ccc-b89c-4b361c4ccc13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:54:32.873032 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.872896    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-tmp-dir\") pod \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") "
Oct 29 23:54:32.873032 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.872935    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-home\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-home\") pod \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") "
Oct 29 23:54:32.873787 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.873756    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-tmp-dir" pod "b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" (UID: "b8ada4d0-3756-4ccc-b89c-4b361c4ccc13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:54:32.873944 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.873863    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-home" pod "b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" (UID: "b8ada4d0-3756-4ccc-b89c-4b361c4ccc13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:54:32.874888 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.874857    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-kube-api-access-znc5d\" (UniqueName: \"kubernetes.io/projected/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-kube-api-access-znc5d\") pod \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") "
Oct 29 23:54:32.875022 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.874935    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-dshm\") pod \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") "
Oct 29 23:54:32.875145 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.875127    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-tls-certs\") pod \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\" (UID: \"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13\") "
Oct 29 23:54:32.876232 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.876152    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-kube-api-access-znc5d" pod "b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" (UID: "b8ada4d0-3756-4ccc-b89c-4b361c4ccc13"). InnerVolumeSpecName "kube-api-access-znc5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:54:32.876413 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.876387    5428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-tmp-dir\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:32.876488 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.876417    5428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-home\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:32.876488 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.876432    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znc5d\" (UniqueName: \"kubernetes.io/projected/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-kube-api-access-znc5d\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:32.876488 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.876446    5428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-model-cache\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:32.879955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.879927    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-dshm" pod "b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" (UID: "b8ada4d0-3756-4ccc-b89c-4b361c4ccc13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:54:32.879955 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.879935    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-tls-certs" pod "b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" (UID: "b8ada4d0-3756-4ccc-b89c-4b361c4ccc13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:54:32.977456 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.977423    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-model-cache\") pod \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") "
Oct 29 23:54:32.977656 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.977467    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d18c7a0-ce12-453d-88cf-7f8de30371ac-kube-api-access-fsdxs\" (UniqueName: \"kubernetes.io/projected/9d18c7a0-ce12-453d-88cf-7f8de30371ac-kube-api-access-fsdxs\") pod \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") "
Oct 29 23:54:32.977656 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.977511    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d18c7a0-ce12-453d-88cf-7f8de30371ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d18c7a0-ce12-453d-88cf-7f8de30371ac-tls-certs\") pod \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") "
Oct 29 23:54:32.977794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.977740    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-model-cache" pod "9d18c7a0-ce12-453d-88cf-7f8de30371ac" (UID: "9d18c7a0-ce12-453d-88cf-7f8de30371ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:54:32.977794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.977756    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-dshm\") pod \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") "
Oct 29 23:54:32.977923 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.977867    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-home\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-home\") pod \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") "
Oct 29 23:54:32.977923 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.977913    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-tmp-dir\") pod \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\" (UID: \"9d18c7a0-ce12-453d-88cf-7f8de30371ac\") "
Oct 29 23:54:32.978229 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.978172    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-home" pod "9d18c7a0-ce12-453d-88cf-7f8de30371ac" (UID: "9d18c7a0-ce12-453d-88cf-7f8de30371ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:54:32.978387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.978243    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-tmp-dir" pod "9d18c7a0-ce12-453d-88cf-7f8de30371ac" (UID: "9d18c7a0-ce12-453d-88cf-7f8de30371ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:54:32.978387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.978353    5428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-home\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:32.978387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.978371    5428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-tmp-dir\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:32.978387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.978381    5428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-model-cache\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:32.978649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.978392    5428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-dshm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:32.978649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.978403    5428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13-tls-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:32.979702 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.979681    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d18c7a0-ce12-453d-88cf-7f8de30371ac-tls-certs" pod "9d18c7a0-ce12-453d-88cf-7f8de30371ac" (UID: "9d18c7a0-ce12-453d-88cf-7f8de30371ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:54:32.979830 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.979699    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-dshm" pod "9d18c7a0-ce12-453d-88cf-7f8de30371ac" (UID: "9d18c7a0-ce12-453d-88cf-7f8de30371ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:54:32.979830 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:32.979748    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d18c7a0-ce12-453d-88cf-7f8de30371ac-kube-api-access-fsdxs" pod "9d18c7a0-ce12-453d-88cf-7f8de30371ac" (UID: "9d18c7a0-ce12-453d-88cf-7f8de30371ac"). InnerVolumeSpecName "kube-api-access-fsdxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:54:33.043193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.043110    5428 generic.go:356] "Generic (PLEG): container finished" podID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerID="dc51cb2a3dfcb15cd3f362c831edb227ba0ee579bc12a8ba57f2386b12211073" exitCode=0
Oct 29 23:54:33.043361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.043289    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"
Oct 29 23:54:33.043361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.043297    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" event={"ID":"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13","Type":"ContainerDied","Data":"dc51cb2a3dfcb15cd3f362c831edb227ba0ee579bc12a8ba57f2386b12211073"}
Oct 29 23:54:33.043361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.043338    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" event={"ID":"b8ada4d0-3756-4ccc-b89c-4b361c4ccc13","Type":"ContainerDied","Data":"fad1a653543d36a80c9de02579dfa7cc189edbe17c23d921e87889376f600cce"}
Oct 29 23:54:33.043361 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.043359    5428 scope.go:122] "RemoveContainer" containerID="dc51cb2a3dfcb15cd3f362c831edb227ba0ee579bc12a8ba57f2386b12211073"
Oct 29 23:54:33.044971 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.044949    5428 generic.go:356] "Generic (PLEG): container finished" podID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerID="61dde9e7557330a23f0882a0c7b222aec7cf5a8538d053a85aa64435afc892e2" exitCode=0
Oct 29 23:54:33.045081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.044973    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" event={"ID":"9d18c7a0-ce12-453d-88cf-7f8de30371ac","Type":"ContainerDied","Data":"61dde9e7557330a23f0882a0c7b222aec7cf5a8538d053a85aa64435afc892e2"}
Oct 29 23:54:33.045081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.045006    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr" event={"ID":"9d18c7a0-ce12-453d-88cf-7f8de30371ac","Type":"ContainerDied","Data":"ab7f05ad2666c0dde2666308e23dad1a06ccf2fe041d4288efb0d31c3bfb4f9d"}
Oct 29 23:54:33.045081 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.045023    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"
Oct 29 23:54:33.053498 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.053478    5428 scope.go:122] "RemoveContainer" containerID="dc51cb2a3dfcb15cd3f362c831edb227ba0ee579bc12a8ba57f2386b12211073"
Oct 29 23:54:33.054027 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:54:33.054004    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc51cb2a3dfcb15cd3f362c831edb227ba0ee579bc12a8ba57f2386b12211073\": container with ID starting with dc51cb2a3dfcb15cd3f362c831edb227ba0ee579bc12a8ba57f2386b12211073 not found: ID does not exist" containerID="dc51cb2a3dfcb15cd3f362c831edb227ba0ee579bc12a8ba57f2386b12211073"
Oct 29 23:54:33.054148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.054039    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc51cb2a3dfcb15cd3f362c831edb227ba0ee579bc12a8ba57f2386b12211073"} err="failed to get container status \"dc51cb2a3dfcb15cd3f362c831edb227ba0ee579bc12a8ba57f2386b12211073\": rpc error: code = NotFound desc = could not find container \"dc51cb2a3dfcb15cd3f362c831edb227ba0ee579bc12a8ba57f2386b12211073\": container with ID starting with dc51cb2a3dfcb15cd3f362c831edb227ba0ee579bc12a8ba57f2386b12211073 not found: ID does not exist"
Oct 29 23:54:33.054148 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.054064    5428 scope.go:122] "RemoveContainer" containerID="61dde9e7557330a23f0882a0c7b222aec7cf5a8538d053a85aa64435afc892e2"
Oct 29 23:54:33.063466 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.063449    5428 scope.go:122] "RemoveContainer" containerID="61dde9e7557330a23f0882a0c7b222aec7cf5a8538d053a85aa64435afc892e2"
Oct 29 23:54:33.063734 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:54:33.063694    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61dde9e7557330a23f0882a0c7b222aec7cf5a8538d053a85aa64435afc892e2\": container with ID starting with 61dde9e7557330a23f0882a0c7b222aec7cf5a8538d053a85aa64435afc892e2 not found: ID does not exist" containerID="61dde9e7557330a23f0882a0c7b222aec7cf5a8538d053a85aa64435afc892e2"
Oct 29 23:54:33.063734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.063720    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61dde9e7557330a23f0882a0c7b222aec7cf5a8538d053a85aa64435afc892e2"} err="failed to get container status \"61dde9e7557330a23f0882a0c7b222aec7cf5a8538d053a85aa64435afc892e2\": rpc error: code = NotFound desc = could not find container \"61dde9e7557330a23f0882a0c7b222aec7cf5a8538d053a85aa64435afc892e2\": container with ID starting with 61dde9e7557330a23f0882a0c7b222aec7cf5a8538d053a85aa64435afc892e2 not found: ID does not exist"
Oct 29 23:54:33.079474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.079446    5428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d18c7a0-ce12-453d-88cf-7f8de30371ac-dshm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:33.079474 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.079473    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsdxs\" (UniqueName: \"kubernetes.io/projected/9d18c7a0-ce12-453d-88cf-7f8de30371ac-kube-api-access-fsdxs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:33.079630 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.079494    5428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d18c7a0-ce12-453d-88cf-7f8de30371ac-tls-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:33.095749 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.095697    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"]
Oct 29 23:54:33.098967 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.098945    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/premium-w3-32c192-kserve-5c5d6b698f-5gqnr"]
Oct 29 23:54:33.115016 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.114993    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:33.132540 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.132521    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"]
Oct 29 23:54:33.137766 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.137749    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g"]
Oct 29 23:54:33.740446 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:33.740406    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-32c192/distinct-w3-32c192-kserve-6bc48b64dd-vkk6g" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:54:34.263188 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:34.263150    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:34.776346 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:34.776309    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" path="/var/lib/kubelet/pods/9d18c7a0-ce12-453d-88cf-7f8de30371ac/volumes"
Oct 29 23:54:34.776752 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:34.776737    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" path="/var/lib/kubelet/pods/b8ada4d0-3756-4ccc-b89c-4b361c4ccc13/volumes"
Oct 29 23:54:34.893135 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:34.893102    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:34.977778 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:34.977734    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp" podUID="b73ad056-0a89-4c23-84b7-d0eceab57c0b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:54:35.243998 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:35.243945    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:36.506876 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:36.506832    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:36.625485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:36.625437    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:36.741902 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:36.741867    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:36.797765 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:36.797685    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"]
Oct 29 23:54:36.797964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:36.797924    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v" podUID="9e6cfa64-3adb-4193-8340-e315e0bbb14e" containerName="istio-proxy" containerID="cri-o://d4ca29e35a2af76806e80598b07d1d844a890d002730126f023cc952d5930920" gracePeriod=30
Oct 29 23:54:36.867735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:36.867693    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:38.576381 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:38.575466    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:38.660073 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:38.659998    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp"
Oct 29 23:54:38.735334 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:38.735300    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b73ad056-0a89-4c23-84b7-d0eceab57c0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73ad056-0a89-4c23-84b7-d0eceab57c0b-plugins-config-volume\") pod \"b73ad056-0a89-4c23-84b7-d0eceab57c0b\" (UID: \"b73ad056-0a89-4c23-84b7-d0eceab57c0b\") "
Oct 29 23:54:38.735531 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:38.735430    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73ad056-0a89-4c23-84b7-d0eceab57c0b-kube-api-access-hv29r\" (UniqueName: \"kubernetes.io/projected/b73ad056-0a89-4c23-84b7-d0eceab57c0b-kube-api-access-hv29r\") pod \"b73ad056-0a89-4c23-84b7-d0eceab57c0b\" (UID: \"b73ad056-0a89-4c23-84b7-d0eceab57c0b\") "
Oct 29 23:54:38.735735 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:38.735710    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73ad056-0a89-4c23-84b7-d0eceab57c0b-plugins-config-volume" pod "b73ad056-0a89-4c23-84b7-d0eceab57c0b" (UID: "b73ad056-0a89-4c23-84b7-d0eceab57c0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:54:38.737697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:38.737666    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73ad056-0a89-4c23-84b7-d0eceab57c0b-kube-api-access-hv29r" pod "b73ad056-0a89-4c23-84b7-d0eceab57c0b" (UID: "b73ad056-0a89-4c23-84b7-d0eceab57c0b"). InnerVolumeSpecName "kube-api-access-hv29r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:54:38.837024 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:38.836983    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv29r\" (UniqueName: \"kubernetes.io/projected/b73ad056-0a89-4c23-84b7-d0eceab57c0b-kube-api-access-hv29r\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:38.837024 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:38.837012    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73ad056-0a89-4c23-84b7-d0eceab57c0b-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:38.851072 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:38.851044    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:39.084867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:39.084771    5428 generic.go:356] "Generic (PLEG): container finished" podID="b73ad056-0a89-4c23-84b7-d0eceab57c0b" containerID="a55be33c86914b092d49c10b843ab03c9e2ff7abe48cb092c1b69dbdd10eb2ae" exitCode=0
Oct 29 23:54:39.084867 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:39.084836    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp"
Oct 29 23:54:39.085107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:39.084874    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp" event={"ID":"b73ad056-0a89-4c23-84b7-d0eceab57c0b","Type":"ContainerDied","Data":"a55be33c86914b092d49c10b843ab03c9e2ff7abe48cb092c1b69dbdd10eb2ae"}
Oct 29 23:54:39.085107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:39.084922    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp" event={"ID":"b73ad056-0a89-4c23-84b7-d0eceab57c0b","Type":"ContainerDied","Data":"0434f934ca2361be0ebe460a9fff4d492a32c14d1799824502ae1af38654a2a0"}
Oct 29 23:54:39.085107 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:39.084943    5428 scope.go:122] "RemoveContainer" containerID="a55be33c86914b092d49c10b843ab03c9e2ff7abe48cb092c1b69dbdd10eb2ae"
Oct 29 23:54:39.095455 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:39.095438    5428 scope.go:122] "RemoveContainer" containerID="a55be33c86914b092d49c10b843ab03c9e2ff7abe48cb092c1b69dbdd10eb2ae"
Oct 29 23:54:39.095753 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:54:39.095733    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a55be33c86914b092d49c10b843ab03c9e2ff7abe48cb092c1b69dbdd10eb2ae\": container with ID starting with a55be33c86914b092d49c10b843ab03c9e2ff7abe48cb092c1b69dbdd10eb2ae not found: ID does not exist" containerID="a55be33c86914b092d49c10b843ab03c9e2ff7abe48cb092c1b69dbdd10eb2ae"
Oct 29 23:54:39.095842 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:39.095761    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a55be33c86914b092d49c10b843ab03c9e2ff7abe48cb092c1b69dbdd10eb2ae"} err="failed to get container status \"a55be33c86914b092d49c10b843ab03c9e2ff7abe48cb092c1b69dbdd10eb2ae\": rpc error: code = NotFound desc = could not find container \"a55be33c86914b092d49c10b843ab03c9e2ff7abe48cb092c1b69dbdd10eb2ae\": container with ID starting with a55be33c86914b092d49c10b843ab03c9e2ff7abe48cb092c1b69dbdd10eb2ae not found: ID does not exist"
Oct 29 23:54:39.136376 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:39.136342    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp"]
Oct 29 23:54:39.149140 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:39.149108    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2126fb-58858c44d8-v8fxp"]
Oct 29 23:54:40.380566 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:40.380533    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:40.775956 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:40.775878    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b73ad056-0a89-4c23-84b7-d0eceab57c0b" path="/var/lib/kubelet/pods/b73ad056-0a89-4c23-84b7-d0eceab57c0b/volumes"
Oct 29 23:54:41.984454 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:41.984414    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2"]
Oct 29 23:54:42.063589 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.063559    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 23:54:42.074529 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.074393    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:42.107038 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.107003    5428 generic.go:356] "Generic (PLEG): container finished" podID="9e6cfa64-3adb-4193-8340-e315e0bbb14e" containerID="d4ca29e35a2af76806e80598b07d1d844a890d002730126f023cc952d5930920" exitCode=0
Oct 29 23:54:42.107233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.107077    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"
Oct 29 23:54:42.107233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.107123    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v" event={"ID":"9e6cfa64-3adb-4193-8340-e315e0bbb14e","Type":"ContainerDied","Data":"d4ca29e35a2af76806e80598b07d1d844a890d002730126f023cc952d5930920"}
Oct 29 23:54:42.107233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.107172    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v" event={"ID":"9e6cfa64-3adb-4193-8340-e315e0bbb14e","Type":"ContainerDied","Data":"bd44276229f4c553c42dbbea0392cd71a11c07aee7c12ea5ac9dada32b63f591"}
Oct 29 23:54:42.107233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.107202    5428 scope.go:122] "RemoveContainer" containerID="d4ca29e35a2af76806e80598b07d1d844a890d002730126f023cc952d5930920"
Oct 29 23:54:42.124762 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.124645    5428 scope.go:122] "RemoveContainer" containerID="d4ca29e35a2af76806e80598b07d1d844a890d002730126f023cc952d5930920"
Oct 29 23:54:42.125097 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:54:42.125019    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ca29e35a2af76806e80598b07d1d844a890d002730126f023cc952d5930920\": container with ID starting with d4ca29e35a2af76806e80598b07d1d844a890d002730126f023cc952d5930920 not found: ID does not exist" containerID="d4ca29e35a2af76806e80598b07d1d844a890d002730126f023cc952d5930920"
Oct 29 23:54:42.125097 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.125061    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ca29e35a2af76806e80598b07d1d844a890d002730126f023cc952d5930920"} err="failed to get container status \"d4ca29e35a2af76806e80598b07d1d844a890d002730126f023cc952d5930920\": rpc error: code = NotFound desc = could not find container \"d4ca29e35a2af76806e80598b07d1d844a890d002730126f023cc952d5930920\": container with ID starting with d4ca29e35a2af76806e80598b07d1d844a890d002730126f023cc952d5930920 not found: ID does not exist"
Oct 29 23:54:42.172559 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.172525    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istiod-ca-cert\") pod \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") "
Oct 29 23:54:42.172721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.172572    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-credential-socket\") pod \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") "
Oct 29 23:54:42.172721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.172602    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-podinfo\") pod \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") "
Oct 29 23:54:42.172721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.172638    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-token\" (UniqueName: \"kubernetes.io/projected/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-token\") pod \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") "
Oct 29 23:54:42.172721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.172670    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-workload-socket\") pod \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") "
Oct 29 23:54:42.172721 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.172706    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-data\") pod \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") "
Oct 29 23:54:42.173062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.172758    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-workload-certs\") pod \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") "
Oct 29 23:54:42.173062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.172832    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-envoy\") pod \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") "
Oct 29 23:54:42.173062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.172918    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6cfa64-3adb-4193-8340-e315e0bbb14e-kube-api-access-lxhz6\" (UniqueName: \"kubernetes.io/projected/9e6cfa64-3adb-4193-8340-e315e0bbb14e-kube-api-access-lxhz6\") pod \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\" (UID: \"9e6cfa64-3adb-4193-8340-e315e0bbb14e\") "
Oct 29 23:54:42.173062 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.172979    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-credential-socket" pod "9e6cfa64-3adb-4193-8340-e315e0bbb14e" (UID: "9e6cfa64-3adb-4193-8340-e315e0bbb14e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:54:42.173273 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.173015    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istiod-ca-cert" pod "9e6cfa64-3adb-4193-8340-e315e0bbb14e" (UID: "9e6cfa64-3adb-4193-8340-e315e0bbb14e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:54:42.173273 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.173087    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-workload-socket" pod "9e6cfa64-3adb-4193-8340-e315e0bbb14e" (UID: "9e6cfa64-3adb-4193-8340-e315e0bbb14e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:54:42.173407 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.173387    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:42.173443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.173413    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:42.173443 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.173428    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:42.173773 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.173747    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-workload-certs" pod "9e6cfa64-3adb-4193-8340-e315e0bbb14e" (UID: "9e6cfa64-3adb-4193-8340-e315e0bbb14e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:54:42.173907 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.173782    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-data" pod "9e6cfa64-3adb-4193-8340-e315e0bbb14e" (UID: "9e6cfa64-3adb-4193-8340-e315e0bbb14e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:54:42.174996 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.174962    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-token" pod "9e6cfa64-3adb-4193-8340-e315e0bbb14e" (UID: "9e6cfa64-3adb-4193-8340-e315e0bbb14e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:54:42.175105 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.174995    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-envoy" pod "9e6cfa64-3adb-4193-8340-e315e0bbb14e" (UID: "9e6cfa64-3adb-4193-8340-e315e0bbb14e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:54:42.175105 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.175006    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-podinfo" pod "9e6cfa64-3adb-4193-8340-e315e0bbb14e" (UID: "9e6cfa64-3adb-4193-8340-e315e0bbb14e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:54:42.175530 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.175509    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6cfa64-3adb-4193-8340-e315e0bbb14e-kube-api-access-lxhz6" pod "9e6cfa64-3adb-4193-8340-e315e0bbb14e" (UID: "9e6cfa64-3adb-4193-8340-e315e0bbb14e"). InnerVolumeSpecName "kube-api-access-lxhz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:54:42.195609 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.195587    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:42.274036 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.273996    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:42.274036 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.274023    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:42.274036 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.274033    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:42.274036 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.274040    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:42.274036 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.274048    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e6cfa64-3adb-4193-8340-e315e0bbb14e-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:42.274405 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.274056    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxhz6\" (UniqueName: \"kubernetes.io/projected/9e6cfa64-3adb-4193-8340-e315e0bbb14e-kube-api-access-lxhz6\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:42.454091 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.454055    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"]
Oct 29 23:54:42.457962 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.457939    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2077b4-openshift-default-66b8f76595-dvx5v"]
Oct 29 23:54:42.775663 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:42.775587    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e6cfa64-3adb-4193-8340-e315e0bbb14e" path="/var/lib/kubelet/pods/9e6cfa64-3adb-4193-8340-e315e0bbb14e/volumes"
Oct 29 23:54:43.518946 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.518902    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"]
Oct 29 23:54:43.519444 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.519157    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t" podUID="002743d1-e4e6-43f4-859e-0bcf233b3179" containerName="maas-api" containerID="cri-o://ec7eb55b24d664da026bde861e2ba00e3e3cc0a7e2b8e4b30de7938b938913ec" gracePeriod=30
Oct 29 23:54:43.589822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.589748    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:43.774898 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.774831    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 23:54:43.892469 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.892436    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/002743d1-e4e6-43f4-859e-0bcf233b3179-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002743d1-e4e6-43f4-859e-0bcf233b3179-openshift-service-ca\") pod \"002743d1-e4e6-43f4-859e-0bcf233b3179\" (UID: \"002743d1-e4e6-43f4-859e-0bcf233b3179\") "
Oct 29 23:54:43.892664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.892480    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/002743d1-e4e6-43f4-859e-0bcf233b3179-metrics-certs\" (UniqueName: \"kubernetes.io/secret/002743d1-e4e6-43f4-859e-0bcf233b3179-metrics-certs\") pod \"002743d1-e4e6-43f4-859e-0bcf233b3179\" (UID: \"002743d1-e4e6-43f4-859e-0bcf233b3179\") "
Oct 29 23:54:43.892664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.892513    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/002743d1-e4e6-43f4-859e-0bcf233b3179-kube-api-access-stbfx\" (UniqueName: \"kubernetes.io/projected/002743d1-e4e6-43f4-859e-0bcf233b3179-kube-api-access-stbfx\") pod \"002743d1-e4e6-43f4-859e-0bcf233b3179\" (UID: \"002743d1-e4e6-43f4-859e-0bcf233b3179\") "
Oct 29 23:54:43.892664 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.892582    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/002743d1-e4e6-43f4-859e-0bcf233b3179-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002743d1-e4e6-43f4-859e-0bcf233b3179-maas-api-tls\") pod \"002743d1-e4e6-43f4-859e-0bcf233b3179\" (UID: \"002743d1-e4e6-43f4-859e-0bcf233b3179\") "
Oct 29 23:54:43.892925 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.892903    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002743d1-e4e6-43f4-859e-0bcf233b3179-openshift-service-ca" pod "002743d1-e4e6-43f4-859e-0bcf233b3179" (UID: "002743d1-e4e6-43f4-859e-0bcf233b3179"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:54:43.893008 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.893000    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002743d1-e4e6-43f4-859e-0bcf233b3179-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:43.894847 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.894787    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/002743d1-e4e6-43f4-859e-0bcf233b3179-maas-api-tls" pod "002743d1-e4e6-43f4-859e-0bcf233b3179" (UID: "002743d1-e4e6-43f4-859e-0bcf233b3179"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:54:43.894961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.894885    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/002743d1-e4e6-43f4-859e-0bcf233b3179-metrics-certs" pod "002743d1-e4e6-43f4-859e-0bcf233b3179" (UID: "002743d1-e4e6-43f4-859e-0bcf233b3179"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:54:43.895015 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.894954    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/002743d1-e4e6-43f4-859e-0bcf233b3179-kube-api-access-stbfx" pod "002743d1-e4e6-43f4-859e-0bcf233b3179" (UID: "002743d1-e4e6-43f4-859e-0bcf233b3179"). InnerVolumeSpecName "kube-api-access-stbfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:54:43.907736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.907701    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"]
Oct 29 23:54:43.907987 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.907960    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x" podUID="d1c40c21-ebcf-4978-bc62-928061baf9b2" containerName="maas-api" containerID="cri-o://c57c51e438bf3bd37673c5dd28e26d6ef11bd14610734f026b954de6db2356c9" gracePeriod=30
Oct 29 23:54:43.987355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.987322    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:43.993769 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.993742    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002743d1-e4e6-43f4-859e-0bcf233b3179-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:43.993890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.993774    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stbfx\" (UniqueName: \"kubernetes.io/projected/002743d1-e4e6-43f4-859e-0bcf233b3179-kube-api-access-stbfx\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:43.993890 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:43.993785    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002743d1-e4e6-43f4-859e-0bcf233b3179-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:44.123997 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.123961    5428 generic.go:356] "Generic (PLEG): container finished" podID="002743d1-e4e6-43f4-859e-0bcf233b3179" containerID="ec7eb55b24d664da026bde861e2ba00e3e3cc0a7e2b8e4b30de7938b938913ec" exitCode=0
Oct 29 23:54:44.124173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.124016    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"
Oct 29 23:54:44.124173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.124097    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t" event={"ID":"002743d1-e4e6-43f4-859e-0bcf233b3179","Type":"ContainerDied","Data":"ec7eb55b24d664da026bde861e2ba00e3e3cc0a7e2b8e4b30de7938b938913ec"}
Oct 29 23:54:44.124173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.124138    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t" event={"ID":"002743d1-e4e6-43f4-859e-0bcf233b3179","Type":"ContainerDied","Data":"4221bc1230c5850bd8f13aa00aa59ab51ebb8e8231308dd27a1eacdf93791b28"}
Oct 29 23:54:44.124173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.124165    5428 scope.go:122] "RemoveContainer" containerID="ec7eb55b24d664da026bde861e2ba00e3e3cc0a7e2b8e4b30de7938b938913ec"
Oct 29 23:54:44.130144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.127694    5428 generic.go:356] "Generic (PLEG): container finished" podID="d1c40c21-ebcf-4978-bc62-928061baf9b2" containerID="c57c51e438bf3bd37673c5dd28e26d6ef11bd14610734f026b954de6db2356c9" exitCode=0
Oct 29 23:54:44.130144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.128063    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x" event={"ID":"d1c40c21-ebcf-4978-bc62-928061baf9b2","Type":"ContainerDied","Data":"c57c51e438bf3bd37673c5dd28e26d6ef11bd14610734f026b954de6db2356c9"}
Oct 29 23:54:44.161041 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.159405    5428 scope.go:122] "RemoveContainer" containerID="ec7eb55b24d664da026bde861e2ba00e3e3cc0a7e2b8e4b30de7938b938913ec"
Oct 29 23:54:44.161041 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:54:44.159932    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec7eb55b24d664da026bde861e2ba00e3e3cc0a7e2b8e4b30de7938b938913ec\": container with ID starting with ec7eb55b24d664da026bde861e2ba00e3e3cc0a7e2b8e4b30de7938b938913ec not found: ID does not exist" containerID="ec7eb55b24d664da026bde861e2ba00e3e3cc0a7e2b8e4b30de7938b938913ec"
Oct 29 23:54:44.161041 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.159969    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec7eb55b24d664da026bde861e2ba00e3e3cc0a7e2b8e4b30de7938b938913ec"} err="failed to get container status \"ec7eb55b24d664da026bde861e2ba00e3e3cc0a7e2b8e4b30de7938b938913ec\": rpc error: code = NotFound desc = could not find container \"ec7eb55b24d664da026bde861e2ba00e3e3cc0a7e2b8e4b30de7938b938913ec\": container with ID starting with ec7eb55b24d664da026bde861e2ba00e3e3cc0a7e2b8e4b30de7938b938913ec not found: ID does not exist"
Oct 29 23:54:44.192485 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.192454    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:54:44.193183 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.193146    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:44.206453 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.206431    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"]
Oct 29 23:54:44.213697 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.213674    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-32c192-6b9695cc78-cqm2t"]
Oct 29 23:54:44.296067 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.295988    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-metrics-certs\") pod \"d1c40c21-ebcf-4978-bc62-928061baf9b2\" (UID: \"d1c40c21-ebcf-4978-bc62-928061baf9b2\") "
Oct 29 23:54:44.296228 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.296083    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c40c21-ebcf-4978-bc62-928061baf9b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c40c21-ebcf-4978-bc62-928061baf9b2-openshift-service-ca\") pod \"d1c40c21-ebcf-4978-bc62-928061baf9b2\" (UID: \"d1c40c21-ebcf-4978-bc62-928061baf9b2\") "
Oct 29 23:54:44.296228 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.296106    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-maas-api-tls\") pod \"d1c40c21-ebcf-4978-bc62-928061baf9b2\" (UID: \"d1c40c21-ebcf-4978-bc62-928061baf9b2\") "
Oct 29 23:54:44.296228 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.296137    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c40c21-ebcf-4978-bc62-928061baf9b2-kube-api-access-hrwjz\" (UniqueName: \"kubernetes.io/projected/d1c40c21-ebcf-4978-bc62-928061baf9b2-kube-api-access-hrwjz\") pod \"d1c40c21-ebcf-4978-bc62-928061baf9b2\" (UID: \"d1c40c21-ebcf-4978-bc62-928061baf9b2\") "
Oct 29 23:54:44.296506 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.296481    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c40c21-ebcf-4978-bc62-928061baf9b2-openshift-service-ca" pod "d1c40c21-ebcf-4978-bc62-928061baf9b2" (UID: "d1c40c21-ebcf-4978-bc62-928061baf9b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:54:44.298055 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.298035    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-metrics-certs" pod "d1c40c21-ebcf-4978-bc62-928061baf9b2" (UID: "d1c40c21-ebcf-4978-bc62-928061baf9b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:54:44.298196 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.298181    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-maas-api-tls" pod "d1c40c21-ebcf-4978-bc62-928061baf9b2" (UID: "d1c40c21-ebcf-4978-bc62-928061baf9b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:54:44.298248 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.298188    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c40c21-ebcf-4978-bc62-928061baf9b2-kube-api-access-hrwjz" pod "d1c40c21-ebcf-4978-bc62-928061baf9b2" (UID: "d1c40c21-ebcf-4978-bc62-928061baf9b2"). InnerVolumeSpecName "kube-api-access-hrwjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:54:44.306935 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.306901    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:44.397493 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.397458    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-maas-api-tls\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:44.397691 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.397498    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrwjz\" (UniqueName: \"kubernetes.io/projected/d1c40c21-ebcf-4978-bc62-928061baf9b2-kube-api-access-hrwjz\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:44.397691 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.397514    5428 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c40c21-ebcf-4978-bc62-928061baf9b2-metrics-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:44.397691 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.397523    5428 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c40c21-ebcf-4978-bc62-928061baf9b2-openshift-service-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:44.566495 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.566417    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:44.684267 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.684231    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:44.775945 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.775911    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="002743d1-e4e6-43f4-859e-0bcf233b3179" path="/var/lib/kubelet/pods/002743d1-e4e6-43f4-859e-0bcf233b3179/volumes"
Oct 29 23:54:44.819317 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.819226    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:44.929599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:44.929569    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:45.135553 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:45.135523    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"
Oct 29 23:54:45.135553 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:45.135537    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x" event={"ID":"d1c40c21-ebcf-4978-bc62-928061baf9b2","Type":"ContainerDied","Data":"af5a0bdf6d5c03ec7465d0b369d3beea34dc0b12da5afdd825ceb1abc8b96846"}
Oct 29 23:54:45.135851 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:45.135601    5428 scope.go:122] "RemoveContainer" containerID="c57c51e438bf3bd37673c5dd28e26d6ef11bd14610734f026b954de6db2356c9"
Oct 29 23:54:45.187523 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:45.187483    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"]
Oct 29 23:54:45.198349 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:45.198317    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-8a8b61-7dfddccfd-htj9x"]
Oct 29 23:54:45.230611 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:45.230585    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:46.583954 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:46.583920    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:46.775433 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:46.775400    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c40c21-ebcf-4978-bc62-928061baf9b2" path="/var/lib/kubelet/pods/d1c40c21-ebcf-4978-bc62-928061baf9b2/volumes"
Oct 29 23:54:46.984820 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:46.984738    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2" podUID="c04a4a59-e364-41e3-bf19-cb3534ad9ec8" containerName="payload-processing" containerID="cri-o://fde35d2819919cef358962e344e972fde9ca5b3ad1869d9a6b718e3307e9a2d6" gracePeriod=25
Oct 29 23:54:47.366601 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:54:47.366569    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:54:48.377942 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:48.377912    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:50.189495 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:50.189458    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:51.369607 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:51.369571    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:51.489144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:51.489114    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:51.751753 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:51.751667    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:51.991315 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:51.991286    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:52.200761 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:52.200727    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:52.319321 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:52.319286    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:52.530145 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:52.530055    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:52.988667 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:52.988631    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:53.240206 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240123    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd"]
Oct 29 23:54:53.240544 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240531    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main"
Oct 29 23:54:53.240589 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240545    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main"
Oct 29 23:54:53.240589 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240556    5428 cpu_manager.go:414] "containerMap: removing container" podUID="9e6cfa64-3adb-4193-8340-e315e0bbb14e" containerName="istio-proxy"
Oct 29 23:54:53.240589 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240563    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e6cfa64-3adb-4193-8340-e315e0bbb14e" containerName="istio-proxy"
Oct 29 23:54:53.240589 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240570    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main"
Oct 29 23:54:53.240589 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240576    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main"
Oct 29 23:54:53.240589 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240589    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b73ad056-0a89-4c23-84b7-d0eceab57c0b" containerName="payload-processing"
Oct 29 23:54:53.240782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240594    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73ad056-0a89-4c23-84b7-d0eceab57c0b" containerName="payload-processing"
Oct 29 23:54:53.240782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240603    5428 cpu_manager.go:414] "containerMap: removing container" podUID="d1c40c21-ebcf-4978-bc62-928061baf9b2" containerName="maas-api"
Oct 29 23:54:53.240782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240608    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c40c21-ebcf-4978-bc62-928061baf9b2" containerName="maas-api"
Oct 29 23:54:53.240782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240625    5428 cpu_manager.go:414] "containerMap: removing container" podUID="002743d1-e4e6-43f4-859e-0bcf233b3179" containerName="maas-api"
Oct 29 23:54:53.240782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240630    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="002743d1-e4e6-43f4-859e-0bcf233b3179" containerName="maas-api"
Oct 29 23:54:53.240782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240640    5428 cpu_manager.go:414] "containerMap: removing container" podUID="84aa240d-42ed-43c5-a84b-001a707311ba" containerName="revoke-keys"
Oct 29 23:54:53.240782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240645    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84aa240d-42ed-43c5-a84b-001a707311ba" containerName="revoke-keys"
Oct 29 23:54:53.240782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240701    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73ad056-0a89-4c23-84b7-d0eceab57c0b" containerName="payload-processing"
Oct 29 23:54:53.240782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240710    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c40c21-ebcf-4978-bc62-928061baf9b2" containerName="maas-api"
Oct 29 23:54:53.240782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240718    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="002743d1-e4e6-43f4-859e-0bcf233b3179" containerName="maas-api"
Oct 29 23:54:53.240782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240726    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="84aa240d-42ed-43c5-a84b-001a707311ba" containerName="revoke-keys"
Oct 29 23:54:53.240782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240733    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d18c7a0-ce12-453d-88cf-7f8de30371ac" containerName="main"
Oct 29 23:54:53.240782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240739    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e6cfa64-3adb-4193-8340-e315e0bbb14e" containerName="istio-proxy"
Oct 29 23:54:53.240782 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.240745    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ada4d0-3756-4ccc-b89c-4b361c4ccc13" containerName="main"
Oct 29 23:54:53.245318 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.245299    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd"
Oct 29 23:54:53.260913 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.260877    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:54:53.263687 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.263658    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd"]
Oct 29 23:54:53.271512 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.271462    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:54:53.271660 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.271639    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:54:53.271723 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.271657    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\""
Oct 29 23:54:53.278901 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.278876    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd\" (UID: \"b63723b8-8d13-43f4-bf4c-a41bf58ca9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd"
Oct 29 23:54:53.279065 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.279042    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd\" (UID: \"b63723b8-8d13-43f4-bf4c-a41bf58ca9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd"
Oct 29 23:54:53.279193 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.279175    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj7jv\" (UniqueName: \"kubernetes.io/projected/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-kube-api-access-tj7jv\") pod \"maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd\" (UID: \"b63723b8-8d13-43f4-bf4c-a41bf58ca9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd"
Oct 29 23:54:53.380041 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.380005    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd\" (UID: \"b63723b8-8d13-43f4-bf4c-a41bf58ca9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd"
Oct 29 23:54:53.380220 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.380091    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj7jv\" (UniqueName: \"kubernetes.io/projected/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-kube-api-access-tj7jv\") pod \"maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd\" (UID: \"b63723b8-8d13-43f4-bf4c-a41bf58ca9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd"
Oct 29 23:54:53.380220 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.380115    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd\" (UID: \"b63723b8-8d13-43f4-bf4c-a41bf58ca9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd"
Oct 29 23:54:53.380843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.380788    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd\" (UID: \"b63723b8-8d13-43f4-bf4c-a41bf58ca9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd"
Oct 29 23:54:53.382363 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.382344    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd\" (UID: \"b63723b8-8d13-43f4-bf4c-a41bf58ca9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd"
Oct 29 23:54:53.403296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.403266    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj7jv\" (UniqueName: \"kubernetes.io/projected/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-kube-api-access-tj7jv\") pod \"maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd\" (UID: \"b63723b8-8d13-43f4-bf4c-a41bf58ca9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd"
Oct 29 23:54:53.560783 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.560706    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd"
Oct 29 23:54:53.864201 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.864172    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd"]
Oct 29 23:54:53.867210 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:54:53.867179    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63723b8_8d13_43f4_bf4c_a41bf58ca9af.slice/crio-e9faf365b27d4a878a214cfea6503144d67c39c6604976fa38398a54b9b3c9ed WatchSource:0}: Error finding container e9faf365b27d4a878a214cfea6503144d67c39c6604976fa38398a54b9b3c9ed: Status 404 returned error can't find the container with id e9faf365b27d4a878a214cfea6503144d67c39c6604976fa38398a54b9b3c9ed
Oct 29 23:54:53.950932 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:53.950901    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:54.190964 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:54.190937    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd" event={"ID":"b63723b8-8d13-43f4-bf4c-a41bf58ca9af","Type":"ContainerStarted","Data":"70b3a60446d45fb7df3a4880c47a6b22c532415ab0ccc3ceeaf304d99bb721db"}
Oct 29 23:54:54.191223 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:54.190985    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd" event={"ID":"b63723b8-8d13-43f4-bf4c-a41bf58ca9af","Type":"ContainerStarted","Data":"e9faf365b27d4a878a214cfea6503144d67c39c6604976fa38398a54b9b3c9ed"}
Oct 29 23:54:54.239908 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:54.239862    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd" podStartSLOduration=1.239848015 podStartE2EDuration="1.239848015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:54:54.237892229 +0000 UTC m=+4297.580840974" watchObservedRunningTime="2026-10-29 23:54:54.239848015 +0000 UTC m=+4297.582796760"
Oct 29 23:54:54.647946 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:54.647907    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:54.784672 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:54.784641    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2" podUID="c04a4a59-e364-41e3-bf19-cb3534ad9ec8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:54:54.943433 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:54.943350    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:55.110836 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:55.110791    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:55.201414 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:55.201319    5428 generic.go:356] "Generic (PLEG): container finished" podID="b63723b8-8d13-43f4-bf4c-a41bf58ca9af" containerID="70b3a60446d45fb7df3a4880c47a6b22c532415ab0ccc3ceeaf304d99bb721db" exitCode=0
Oct 29 23:54:55.201579 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:55.201407    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd" event={"ID":"b63723b8-8d13-43f4-bf4c-a41bf58ca9af","Type":"ContainerDied","Data":"70b3a60446d45fb7df3a4880c47a6b22c532415ab0ccc3ceeaf304d99bb721db"}
Oct 29 23:54:55.354052 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:55.354017    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:55.855689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:55.855656    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:56.334424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:56.334399    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd"
Oct 29 23:54:56.403118 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:56.403083    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-maas-api-xks-ca\") pod \"b63723b8-8d13-43f4-bf4c-a41bf58ca9af\" (UID: \"b63723b8-8d13-43f4-bf4c-a41bf58ca9af\") "
Oct 29 23:54:56.403313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:56.403142    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-kube-api-access-tj7jv\" (UniqueName: \"kubernetes.io/projected/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-kube-api-access-tj7jv\") pod \"b63723b8-8d13-43f4-bf4c-a41bf58ca9af\" (UID: \"b63723b8-8d13-43f4-bf4c-a41bf58ca9af\") "
Oct 29 23:54:56.403313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:56.403211    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-maas-api-openshift-ca\") pod \"b63723b8-8d13-43f4-bf4c-a41bf58ca9af\" (UID: \"b63723b8-8d13-43f4-bf4c-a41bf58ca9af\") "
Oct 29 23:54:56.403569 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:56.403545    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-maas-api-openshift-ca" pod "b63723b8-8d13-43f4-bf4c-a41bf58ca9af" (UID: "b63723b8-8d13-43f4-bf4c-a41bf58ca9af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:54:56.405256 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:56.405228    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-kube-api-access-tj7jv" pod "b63723b8-8d13-43f4-bf4c-a41bf58ca9af" (UID: "b63723b8-8d13-43f4-bf4c-a41bf58ca9af"). InnerVolumeSpecName "kube-api-access-tj7jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:54:56.405400 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:56.405303    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-maas-api-xks-ca" pod "b63723b8-8d13-43f4-bf4c-a41bf58ca9af" (UID: "b63723b8-8d13-43f4-bf4c-a41bf58ca9af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:54:56.437202 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:56.437170    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:56.504365 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:56.504287    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-maas-api-openshift-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:56.504365 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:56.504323    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-maas-api-xks-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:56.504365 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:56.504333    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj7jv\" (UniqueName: \"kubernetes.io/projected/b63723b8-8d13-43f4-bf4c-a41bf58ca9af-kube-api-access-tj7jv\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:54:57.214216 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:57.214183    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd"
Oct 29 23:54:57.214736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:57.214175    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd" event={"ID":"b63723b8-8d13-43f4-bf4c-a41bf58ca9af","Type":"ContainerDied","Data":"e9faf365b27d4a878a214cfea6503144d67c39c6604976fa38398a54b9b3c9ed"}
Oct 29 23:54:57.214736 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:57.214329    5428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9faf365b27d4a878a214cfea6503144d67c39c6604976fa38398a54b9b3c9ed"
Oct 29 23:54:57.739070 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:57.739041    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:57.949423 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:57.949385    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"]
Oct 29 23:54:58.046713 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:58.046632    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:54:59.493519 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:54:59.493488    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:01.280698 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:01.280660    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:02.535263 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:02.535225    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:02.984749 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:02.984712    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:03.673089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:03.673054    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf"]
Oct 29 23:55:03.787509 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:03.787476    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:03.898333 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:03.898292    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:04.450491 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.450455    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg"]
Oct 29 23:55:04.450923 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.450908    5428 cpu_manager.go:414] "containerMap: removing container" podUID="b63723b8-8d13-43f4-bf4c-a41bf58ca9af" containerName="revoke-keys"
Oct 29 23:55:04.450983 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.450924    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b63723b8-8d13-43f4-bf4c-a41bf58ca9af" containerName="revoke-keys"
Oct 29 23:55:04.451022 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.450990    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="b63723b8-8d13-43f4-bf4c-a41bf58ca9af" containerName="revoke-keys"
Oct 29 23:55:04.454271 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.454245    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg"
Oct 29 23:55:04.474427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.474404    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:55:04.477116 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.477042    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg"]
Oct 29 23:55:04.477237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.477136    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905a49ab-1b16-4584-b8b2-c6eb1e597134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42f63f53-ggszg\" (UID: \"905a49ab-1b16-4584-b8b2-c6eb1e597134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg"
Oct 29 23:55:04.477355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.477336    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9249\" (UniqueName: \"kubernetes.io/projected/905a49ab-1b16-4584-b8b2-c6eb1e597134-kube-api-access-h9249\") pod \"maas-api-revoke-sub-42f63f53-ggszg\" (UID: \"905a49ab-1b16-4584-b8b2-c6eb1e597134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg"
Oct 29 23:55:04.477422 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.477369    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905a49ab-1b16-4584-b8b2-c6eb1e597134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42f63f53-ggszg\" (UID: \"905a49ab-1b16-4584-b8b2-c6eb1e597134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg"
Oct 29 23:55:04.485909 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.485887    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9l4zr\""
Oct 29 23:55:04.486035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.485979    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:55:04.486035 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.485996    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:55:04.578763 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.578724    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9249\" (UniqueName: \"kubernetes.io/projected/905a49ab-1b16-4584-b8b2-c6eb1e597134-kube-api-access-h9249\") pod \"maas-api-revoke-sub-42f63f53-ggszg\" (UID: \"905a49ab-1b16-4584-b8b2-c6eb1e597134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg"
Oct 29 23:55:04.578970 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.578769    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905a49ab-1b16-4584-b8b2-c6eb1e597134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42f63f53-ggszg\" (UID: \"905a49ab-1b16-4584-b8b2-c6eb1e597134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg"
Oct 29 23:55:04.578970 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.578850    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905a49ab-1b16-4584-b8b2-c6eb1e597134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42f63f53-ggszg\" (UID: \"905a49ab-1b16-4584-b8b2-c6eb1e597134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg"
Oct 29 23:55:04.579560 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.579537    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905a49ab-1b16-4584-b8b2-c6eb1e597134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42f63f53-ggszg\" (UID: \"905a49ab-1b16-4584-b8b2-c6eb1e597134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg"
Oct 29 23:55:04.581355 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.581333    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905a49ab-1b16-4584-b8b2-c6eb1e597134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42f63f53-ggszg\" (UID: \"905a49ab-1b16-4584-b8b2-c6eb1e597134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg"
Oct 29 23:55:04.601829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.601788    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9249\" (UniqueName: \"kubernetes.io/projected/905a49ab-1b16-4584-b8b2-c6eb1e597134-kube-api-access-h9249\") pod \"maas-api-revoke-sub-42f63f53-ggszg\" (UID: \"905a49ab-1b16-4584-b8b2-c6eb1e597134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg"
Oct 29 23:55:04.768030 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.767954    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg"
Oct 29 23:55:04.785551 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:04.785520    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2" podUID="c04a4a59-e364-41e3-bf19-cb3534ad9ec8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:55:05.069066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:05.069037    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg"]
Oct 29 23:55:05.071749 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:55:05.071707    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905a49ab_1b16_4584_b8b2_c6eb1e597134.slice/crio-12e09b6198db89bffbb8a88fca1df54c8b829f78d30b6af078dd59ac643a366f WatchSource:0}: Error finding container 12e09b6198db89bffbb8a88fca1df54c8b829f78d30b6af078dd59ac643a366f: Status 404 returned error can't find the container with id 12e09b6198db89bffbb8a88fca1df54c8b829f78d30b6af078dd59ac643a366f
Oct 29 23:55:05.258957 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:05.258916    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg" event={"ID":"905a49ab-1b16-4584-b8b2-c6eb1e597134","Type":"ContainerStarted","Data":"12e09b6198db89bffbb8a88fca1df54c8b829f78d30b6af078dd59ac643a366f"}
Oct 29 23:55:06.266517 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:06.266422    5428 generic.go:356] "Generic (PLEG): container finished" podID="905a49ab-1b16-4584-b8b2-c6eb1e597134" containerID="af7c790d27e447022b94d18813fa315e55cb9305a6d280c0e2bec45bf67ef85b" exitCode=0
Oct 29 23:55:06.266517 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:06.266499    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg" event={"ID":"905a49ab-1b16-4584-b8b2-c6eb1e597134","Type":"ContainerDied","Data":"af7c790d27e447022b94d18813fa315e55cb9305a6d280c0e2bec45bf67ef85b"}
Oct 29 23:55:07.247589 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.247527    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2"
Oct 29 23:55:07.282829 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.282770    5428 generic.go:356] "Generic (PLEG): container finished" podID="c04a4a59-e364-41e3-bf19-cb3534ad9ec8" containerID="fde35d2819919cef358962e344e972fde9ca5b3ad1869d9a6b718e3307e9a2d6" exitCode=0
Oct 29 23:55:07.283264 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.282837    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2" event={"ID":"c04a4a59-e364-41e3-bf19-cb3534ad9ec8","Type":"ContainerDied","Data":"fde35d2819919cef358962e344e972fde9ca5b3ad1869d9a6b718e3307e9a2d6"}
Oct 29 23:55:07.283264 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.282849    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2"
Oct 29 23:55:07.283264 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.282927    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2" event={"ID":"c04a4a59-e364-41e3-bf19-cb3534ad9ec8","Type":"ContainerDied","Data":"fddac037f81293cacb403ae6fbeef17d62551155408f0663009cb2ab92ec985b"}
Oct 29 23:55:07.283264 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.282962    5428 scope.go:122] "RemoveContainer" containerID="fde35d2819919cef358962e344e972fde9ca5b3ad1869d9a6b718e3307e9a2d6"
Oct 29 23:55:07.305246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.305212    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04a4a59-e364-41e3-bf19-cb3534ad9ec8-kube-api-access-5wwdl\" (UniqueName: \"kubernetes.io/projected/c04a4a59-e364-41e3-bf19-cb3534ad9ec8-kube-api-access-5wwdl\") pod \"c04a4a59-e364-41e3-bf19-cb3534ad9ec8\" (UID: \"c04a4a59-e364-41e3-bf19-cb3534ad9ec8\") "
Oct 29 23:55:07.305532 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.305509    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04a4a59-e364-41e3-bf19-cb3534ad9ec8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04a4a59-e364-41e3-bf19-cb3534ad9ec8-plugins-config-volume\") pod \"c04a4a59-e364-41e3-bf19-cb3534ad9ec8\" (UID: \"c04a4a59-e364-41e3-bf19-cb3534ad9ec8\") "
Oct 29 23:55:07.306144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.305910    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04a4a59-e364-41e3-bf19-cb3534ad9ec8-plugins-config-volume" pod "c04a4a59-e364-41e3-bf19-cb3534ad9ec8" (UID: "c04a4a59-e364-41e3-bf19-cb3534ad9ec8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:55:07.306144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.306115    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04a4a59-e364-41e3-bf19-cb3534ad9ec8-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:07.308690 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.308391    5428 scope.go:122] "RemoveContainer" containerID="fde35d2819919cef358962e344e972fde9ca5b3ad1869d9a6b718e3307e9a2d6"
Oct 29 23:55:07.309048 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:55:07.308936    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde35d2819919cef358962e344e972fde9ca5b3ad1869d9a6b718e3307e9a2d6\": container with ID starting with fde35d2819919cef358962e344e972fde9ca5b3ad1869d9a6b718e3307e9a2d6 not found: ID does not exist" containerID="fde35d2819919cef358962e344e972fde9ca5b3ad1869d9a6b718e3307e9a2d6"
Oct 29 23:55:07.309048 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.308972    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde35d2819919cef358962e344e972fde9ca5b3ad1869d9a6b718e3307e9a2d6"} err="failed to get container status \"fde35d2819919cef358962e344e972fde9ca5b3ad1869d9a6b718e3307e9a2d6\": rpc error: code = NotFound desc = could not find container \"fde35d2819919cef358962e344e972fde9ca5b3ad1869d9a6b718e3307e9a2d6\": container with ID starting with fde35d2819919cef358962e344e972fde9ca5b3ad1869d9a6b718e3307e9a2d6 not found: ID does not exist"
Oct 29 23:55:07.314353 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.314327    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04a4a59-e364-41e3-bf19-cb3534ad9ec8-kube-api-access-5wwdl" pod "c04a4a59-e364-41e3-bf19-cb3534ad9ec8" (UID: "c04a4a59-e364-41e3-bf19-cb3534ad9ec8"). InnerVolumeSpecName "kube-api-access-5wwdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:55:07.407630 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.407590    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wwdl\" (UniqueName: \"kubernetes.io/projected/c04a4a59-e364-41e3-bf19-cb3534ad9ec8-kube-api-access-5wwdl\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:07.433941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.433905    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg"
Oct 29 23:55:07.508691 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.508612    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/905a49ab-1b16-4584-b8b2-c6eb1e597134-kube-api-access-h9249\" (UniqueName: \"kubernetes.io/projected/905a49ab-1b16-4584-b8b2-c6eb1e597134-kube-api-access-h9249\") pod \"905a49ab-1b16-4584-b8b2-c6eb1e597134\" (UID: \"905a49ab-1b16-4584-b8b2-c6eb1e597134\") "
Oct 29 23:55:07.508881 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.508738    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/905a49ab-1b16-4584-b8b2-c6eb1e597134-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905a49ab-1b16-4584-b8b2-c6eb1e597134-maas-api-xks-ca\") pod \"905a49ab-1b16-4584-b8b2-c6eb1e597134\" (UID: \"905a49ab-1b16-4584-b8b2-c6eb1e597134\") "
Oct 29 23:55:07.508881 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.508859    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/905a49ab-1b16-4584-b8b2-c6eb1e597134-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905a49ab-1b16-4584-b8b2-c6eb1e597134-maas-api-openshift-ca\") pod \"905a49ab-1b16-4584-b8b2-c6eb1e597134\" (UID: \"905a49ab-1b16-4584-b8b2-c6eb1e597134\") "
Oct 29 23:55:07.509345 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.509311    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/905a49ab-1b16-4584-b8b2-c6eb1e597134-maas-api-openshift-ca" pod "905a49ab-1b16-4584-b8b2-c6eb1e597134" (UID: "905a49ab-1b16-4584-b8b2-c6eb1e597134"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:55:07.510938 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.510912    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905a49ab-1b16-4584-b8b2-c6eb1e597134-kube-api-access-h9249" pod "905a49ab-1b16-4584-b8b2-c6eb1e597134" (UID: "905a49ab-1b16-4584-b8b2-c6eb1e597134"). InnerVolumeSpecName "kube-api-access-h9249". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:55:07.511165 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.511149    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/905a49ab-1b16-4584-b8b2-c6eb1e597134-maas-api-xks-ca" pod "905a49ab-1b16-4584-b8b2-c6eb1e597134" (UID: "905a49ab-1b16-4584-b8b2-c6eb1e597134"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:55:07.610146 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.610117    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905a49ab-1b16-4584-b8b2-c6eb1e597134-maas-api-openshift-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:07.610146 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.610144    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9249\" (UniqueName: \"kubernetes.io/projected/905a49ab-1b16-4584-b8b2-c6eb1e597134-kube-api-access-h9249\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:07.610347 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.610159    5428 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905a49ab-1b16-4584-b8b2-c6eb1e597134-maas-api-xks-ca\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:07.628210 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.628181    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2"]
Oct 29 23:55:07.634200 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:07.634164    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-8a8b61-578cbc78bd-rxvt2"]
Oct 29 23:55:08.290372 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:08.290286    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg"
Oct 29 23:55:08.290372 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:08.290313    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f63f53-ggszg" event={"ID":"905a49ab-1b16-4584-b8b2-c6eb1e597134","Type":"ContainerDied","Data":"12e09b6198db89bffbb8a88fca1df54c8b829f78d30b6af078dd59ac643a366f"}
Oct 29 23:55:08.290922 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:08.290373    5428 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12e09b6198db89bffbb8a88fca1df54c8b829f78d30b6af078dd59ac643a366f"
Oct 29 23:55:08.687823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:08.680878    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf" podUID="2032f74c-64e7-4c12-a969-c5b7eb33eb53" containerName="payload-pre-processing" containerID="cri-o://cf840fcb64477dc0bf276c04b2e55292d05ba147d9b3fff5a21089ccd3b7ef1d" gracePeriod=25
Oct 29 23:55:08.776852 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:08.776790    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04a4a59-e364-41e3-bf19-cb3534ad9ec8" path="/var/lib/kubelet/pods/c04a4a59-e364-41e3-bf19-cb3534ad9ec8/volumes"
Oct 29 23:55:12.969564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:12.969518    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main" containerID="cri-o://34af99d2461bcebede0b4d541f2a4ec07b87cfe75583e99f9e73ac827c1ff7fa" gracePeriod=45
Oct 29 23:55:13.224313 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.224256    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 23:55:13.278666 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.278634    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-tmp-dir" pod "3c62dd36-e795-4778-902e-a990d1998123" (UID: "3c62dd36-e795-4778-902e-a990d1998123"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:55:13.279424 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.279397    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-tmp-dir\") pod \"3c62dd36-e795-4778-902e-a990d1998123\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") "
Oct 29 23:55:13.279549 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.279464    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-dshm\") pod \"3c62dd36-e795-4778-902e-a990d1998123\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") "
Oct 29 23:55:13.279549 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.279502    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c62dd36-e795-4778-902e-a990d1998123-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c62dd36-e795-4778-902e-a990d1998123-tls-certs\") pod \"3c62dd36-e795-4778-902e-a990d1998123\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") "
Oct 29 23:55:13.279549 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.279530    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c62dd36-e795-4778-902e-a990d1998123-kube-api-access-rrdxm\" (UniqueName: \"kubernetes.io/projected/3c62dd36-e795-4778-902e-a990d1998123-kube-api-access-rrdxm\") pod \"3c62dd36-e795-4778-902e-a990d1998123\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") "
Oct 29 23:55:13.279708 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.279602    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-home\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-home\") pod \"3c62dd36-e795-4778-902e-a990d1998123\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") "
Oct 29 23:55:13.279708 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.279640    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-model-cache\") pod \"3c62dd36-e795-4778-902e-a990d1998123\" (UID: \"3c62dd36-e795-4778-902e-a990d1998123\") "
Oct 29 23:55:13.280048 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.280018    5428 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-tmp-dir\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:13.280312 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.280292    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-model-cache" pod "3c62dd36-e795-4778-902e-a990d1998123" (UID: "3c62dd36-e795-4778-902e-a990d1998123"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:55:13.280557 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.280539    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-home" pod "3c62dd36-e795-4778-902e-a990d1998123" (UID: "3c62dd36-e795-4778-902e-a990d1998123"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:55:13.282552 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.282524    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-dshm" pod "3c62dd36-e795-4778-902e-a990d1998123" (UID: "3c62dd36-e795-4778-902e-a990d1998123"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:55:13.282828 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.282772    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c62dd36-e795-4778-902e-a990d1998123-kube-api-access-rrdxm" pod "3c62dd36-e795-4778-902e-a990d1998123" (UID: "3c62dd36-e795-4778-902e-a990d1998123"). InnerVolumeSpecName "kube-api-access-rrdxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:55:13.282920 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.282828    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c62dd36-e795-4778-902e-a990d1998123-tls-certs" pod "3c62dd36-e795-4778-902e-a990d1998123" (UID: "3c62dd36-e795-4778-902e-a990d1998123"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:55:13.323658 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.323617    5428 generic.go:356] "Generic (PLEG): container finished" podID="3c62dd36-e795-4778-902e-a990d1998123" containerID="34af99d2461bcebede0b4d541f2a4ec07b87cfe75583e99f9e73ac827c1ff7fa" exitCode=0
Oct 29 23:55:13.323840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.323722    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" event={"ID":"3c62dd36-e795-4778-902e-a990d1998123","Type":"ContainerDied","Data":"34af99d2461bcebede0b4d541f2a4ec07b87cfe75583e99f9e73ac827c1ff7fa"}
Oct 29 23:55:13.323840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.323763    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk" event={"ID":"3c62dd36-e795-4778-902e-a990d1998123","Type":"ContainerDied","Data":"d08dc7e729173fd949915c2968d1246aec510eef193b206348112a8b95de8a98"}
Oct 29 23:55:13.323840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.323785    5428 scope.go:122] "RemoveContainer" containerID="34af99d2461bcebede0b4d541f2a4ec07b87cfe75583e99f9e73ac827c1ff7fa"
Oct 29 23:55:13.323840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.323725    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"
Oct 29 23:55:13.336843 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.336812    5428 scope.go:122] "RemoveContainer" containerID="34af99d2461bcebede0b4d541f2a4ec07b87cfe75583e99f9e73ac827c1ff7fa"
Oct 29 23:55:13.337167 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:55:13.337142    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34af99d2461bcebede0b4d541f2a4ec07b87cfe75583e99f9e73ac827c1ff7fa\": container with ID starting with 34af99d2461bcebede0b4d541f2a4ec07b87cfe75583e99f9e73ac827c1ff7fa not found: ID does not exist" containerID="34af99d2461bcebede0b4d541f2a4ec07b87cfe75583e99f9e73ac827c1ff7fa"
Oct 29 23:55:13.337233 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.337180    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34af99d2461bcebede0b4d541f2a4ec07b87cfe75583e99f9e73ac827c1ff7fa"} err="failed to get container status \"34af99d2461bcebede0b4d541f2a4ec07b87cfe75583e99f9e73ac827c1ff7fa\": rpc error: code = NotFound desc = could not find container \"34af99d2461bcebede0b4d541f2a4ec07b87cfe75583e99f9e73ac827c1ff7fa\": container with ID starting with 34af99d2461bcebede0b4d541f2a4ec07b87cfe75583e99f9e73ac827c1ff7fa not found: ID does not exist"
Oct 29 23:55:13.375302 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.375270    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"]
Oct 29 23:55:13.379357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.379330    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-99dfb0/distinct-w1-99dfb0-kserve-59b7978874-ngqmk"]
Oct 29 23:55:13.381088 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.381061    5428 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-dshm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:13.381144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.381099    5428 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c62dd36-e795-4778-902e-a990d1998123-tls-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:13.381144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.381116    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrdxm\" (UniqueName: \"kubernetes.io/projected/3c62dd36-e795-4778-902e-a990d1998123-kube-api-access-rrdxm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:13.381144 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.381132    5428 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-home\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:13.381264 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.381149    5428 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c62dd36-e795-4778-902e-a990d1998123-model-cache\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:13.645784 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:13.645745    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf" podUID="2032f74c-64e7-4c12-a969-c5b7eb33eb53" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:55:14.774771 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:14.774726    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c62dd36-e795-4778-902e-a990d1998123" path="/var/lib/kubelet/pods/3c62dd36-e795-4778-902e-a990d1998123/volumes"
Oct 29 23:55:17.368208 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:55:17.368171    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:55:19.652823 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:19.652766    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:20.153362 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:20.153324    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:20.274319 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:20.274287    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:20.276366 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:20.276339    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd"]
Oct 29 23:55:20.287574 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:20.287549    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-99dfb0-lltgd"]
Oct 29 23:55:20.779433 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:20.779389    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b63723b8-8d13-43f4-bf4c-a41bf58ca9af" path="/var/lib/kubelet/pods/b63723b8-8d13-43f4-bf4c-a41bf58ca9af/volumes"
Oct 29 23:55:20.813626 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:20.813593    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:23.645520 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:23.645485    5428 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf" podUID="2032f74c-64e7-4c12-a969-c5b7eb33eb53" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:55:26.805712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:26.805681    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:26.927959 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:26.927923    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:27.105577 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:27.105467    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"]
Oct 29 23:55:27.106357 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:27.106098    5428 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp" podUID="7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" containerName="istio-proxy" containerID="cri-o://aed1b6105d7d4e5f94879c29f415f77abebeaf508b5ea2f7419a6fc1080d94ef" gracePeriod=30
Oct 29 23:55:27.143018 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:27.142988    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:28.165097 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:28.165019    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f48949686-sngj6_96b9b7f3-d372-44ca-adfb-79546be9f85e/authorino/0.log"
Oct 29 23:55:28.392812 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:28.392781    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-dqxbf_7f6616af-b4f8-4cd9-9cd1-c356b28d97c4/manager/0.log"
Oct 29 23:55:28.863891 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:28.863866    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf"
Oct 29 23:55:29.031201 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:29.031108    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2032f74c-64e7-4c12-a969-c5b7eb33eb53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2032f74c-64e7-4c12-a969-c5b7eb33eb53-plugins-config-volume\") pod \"2032f74c-64e7-4c12-a969-c5b7eb33eb53\" (UID: \"2032f74c-64e7-4c12-a969-c5b7eb33eb53\") "
Oct 29 23:55:29.031201 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:29.031175    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2032f74c-64e7-4c12-a969-c5b7eb33eb53-kube-api-access-zr7ds\" (UniqueName: \"kubernetes.io/projected/2032f74c-64e7-4c12-a969-c5b7eb33eb53-kube-api-access-zr7ds\") pod \"2032f74c-64e7-4c12-a969-c5b7eb33eb53\" (UID: \"2032f74c-64e7-4c12-a969-c5b7eb33eb53\") "
Oct 29 23:55:29.031602 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:29.031575    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2032f74c-64e7-4c12-a969-c5b7eb33eb53-plugins-config-volume" pod "2032f74c-64e7-4c12-a969-c5b7eb33eb53" (UID: "2032f74c-64e7-4c12-a969-c5b7eb33eb53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:55:29.033489 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:29.033466    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2032f74c-64e7-4c12-a969-c5b7eb33eb53-kube-api-access-zr7ds" pod "2032f74c-64e7-4c12-a969-c5b7eb33eb53" (UID: "2032f74c-64e7-4c12-a969-c5b7eb33eb53"). InnerVolumeSpecName "kube-api-access-zr7ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:55:29.132301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:29.132266    5428 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2032f74c-64e7-4c12-a969-c5b7eb33eb53-plugins-config-volume\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:29.132301 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:29.132301    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr7ds\" (UniqueName: \"kubernetes.io/projected/2032f74c-64e7-4c12-a969-c5b7eb33eb53-kube-api-access-zr7ds\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:29.432961 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:29.432925    5428 generic.go:356] "Generic (PLEG): container finished" podID="2032f74c-64e7-4c12-a969-c5b7eb33eb53" containerID="cf840fcb64477dc0bf276c04b2e55292d05ba147d9b3fff5a21089ccd3b7ef1d" exitCode=0
Oct 29 23:55:29.433416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:29.432974    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf"
Oct 29 23:55:29.433416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:29.433016    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf" event={"ID":"2032f74c-64e7-4c12-a969-c5b7eb33eb53","Type":"ContainerDied","Data":"cf840fcb64477dc0bf276c04b2e55292d05ba147d9b3fff5a21089ccd3b7ef1d"}
Oct 29 23:55:29.433416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:29.433068    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf" event={"ID":"2032f74c-64e7-4c12-a969-c5b7eb33eb53","Type":"ContainerDied","Data":"364e94cb31bb3eaf93cba75d7af62307820e1c7bd7b56e5b414d0c05ff087d9a"}
Oct 29 23:55:29.433416 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:29.433091    5428 scope.go:122] "RemoveContainer" containerID="cf840fcb64477dc0bf276c04b2e55292d05ba147d9b3fff5a21089ccd3b7ef1d"
Oct 29 23:55:29.443226 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:29.443210    5428 scope.go:122] "RemoveContainer" containerID="cf840fcb64477dc0bf276c04b2e55292d05ba147d9b3fff5a21089ccd3b7ef1d"
Oct 29 23:55:29.443463 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:55:29.443446    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf840fcb64477dc0bf276c04b2e55292d05ba147d9b3fff5a21089ccd3b7ef1d\": container with ID starting with cf840fcb64477dc0bf276c04b2e55292d05ba147d9b3fff5a21089ccd3b7ef1d not found: ID does not exist" containerID="cf840fcb64477dc0bf276c04b2e55292d05ba147d9b3fff5a21089ccd3b7ef1d"
Oct 29 23:55:29.443518 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:29.443473    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf840fcb64477dc0bf276c04b2e55292d05ba147d9b3fff5a21089ccd3b7ef1d"} err="failed to get container status \"cf840fcb64477dc0bf276c04b2e55292d05ba147d9b3fff5a21089ccd3b7ef1d\": rpc error: code = NotFound desc = could not find container \"cf840fcb64477dc0bf276c04b2e55292d05ba147d9b3fff5a21089ccd3b7ef1d\": container with ID starting with cf840fcb64477dc0bf276c04b2e55292d05ba147d9b3fff5a21089ccd3b7ef1d not found: ID does not exist"
Oct 29 23:55:29.483696 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:29.483669    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf"]
Oct 29 23:55:29.495429 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:29.495408    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-99dfb0-6595d5976c-dfbcf"]
Oct 29 23:55:30.782097 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:30.782059    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2032f74c-64e7-4c12-a969-c5b7eb33eb53" path="/var/lib/kubelet/pods/2032f74c-64e7-4c12-a969-c5b7eb33eb53/volumes"
Oct 29 23:55:31.050793 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:31.050713    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7bdc847df-4dfgz_6d23d603-f91c-4a7b-85e0-8c25b786b480/manager/0.log"
Oct 29 23:55:31.273427 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:31.273394    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lz8kp_1ad543b2-b749-40f3-9210-4026141bbf3d/manager/0.log"
Oct 29 23:55:31.481599 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:31.481562    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-6pqtr_6d668198-fafe-4276-9474-e55b192582bf/manager/0.log"
Oct 29 23:55:32.365824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.365786    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 23:55:32.454256 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.454167    5428 generic.go:356] "Generic (PLEG): container finished" podID="7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" containerID="aed1b6105d7d4e5f94879c29f415f77abebeaf508b5ea2f7419a6fc1080d94ef" exitCode=0
Oct 29 23:55:32.454256 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.454214    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp" event={"ID":"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc","Type":"ContainerDied","Data":"aed1b6105d7d4e5f94879c29f415f77abebeaf508b5ea2f7419a6fc1080d94ef"}
Oct 29 23:55:32.454469 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.454267    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp" event={"ID":"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc","Type":"ContainerDied","Data":"6be51cbb85875e2ab45ab83be7cf881d7ea23e5d8e355312fcfc2db95ff48836"}
Oct 29 23:55:32.454469 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.454290    5428 scope.go:122] "RemoveContainer" containerID="aed1b6105d7d4e5f94879c29f415f77abebeaf508b5ea2f7419a6fc1080d94ef"
Oct 29 23:55:32.454469 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.454224    5428 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"
Oct 29 23:55:32.462480 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.462455    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-kube-api-access-c7zgm\" (UniqueName: \"kubernetes.io/projected/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-kube-api-access-c7zgm\") pod \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") "
Oct 29 23:55:32.462608 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.462516    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-token\" (UniqueName: \"kubernetes.io/projected/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-token\") pod \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") "
Oct 29 23:55:32.462608 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.462536    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-credential-socket\") pod \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") "
Oct 29 23:55:32.462709 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.462638    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-podinfo\") pod \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") "
Oct 29 23:55:32.462709 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.462694    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istiod-ca-cert\") pod \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") "
Oct 29 23:55:32.462840 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.462733    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-data\") pod \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") "
Oct 29 23:55:32.462908 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.462864    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-credential-socket" pod "7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" (UID: "7c524b88-93cc-4890-aca0-d1dbd4b2f2dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:55:32.462908 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.462891    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-workload-certs\") pod \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") "
Oct 29 23:55:32.463024 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.462937    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-envoy\") pod \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") "
Oct 29 23:55:32.463024 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.462987    5428 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-workload-socket\") pod \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\" (UID: \"7c524b88-93cc-4890-aca0-d1dbd4b2f2dc\") "
Oct 29 23:55:32.463139 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.463106    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istiod-ca-cert" pod "7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" (UID: "7c524b88-93cc-4890-aca0-d1dbd4b2f2dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:55:32.463516 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.463234    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-workload-certs" pod "7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" (UID: "7c524b88-93cc-4890-aca0-d1dbd4b2f2dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:55:32.463516 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.463343    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-workload-socket" pod "7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" (UID: "7c524b88-93cc-4890-aca0-d1dbd4b2f2dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:55:32.463516 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.463458    5428 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-workload-certs\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:32.463516 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.463472    5428 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-workload-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:32.463516 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.463484    5428 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-credential-socket\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:32.463516 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.463497    5428 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istiod-ca-cert\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:32.464190 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.464168    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-data" pod "7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" (UID: "7c524b88-93cc-4890-aca0-d1dbd4b2f2dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:55:32.464632 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.464614    5428 scope.go:122] "RemoveContainer" containerID="122b37eccf0ed7fcb6d783ae4cb929f99107edc0f7a1a2e5babe95c59fd99637"
Oct 29 23:55:32.465066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.465042    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-kube-api-access-c7zgm" pod "7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" (UID: "7c524b88-93cc-4890-aca0-d1dbd4b2f2dc"). InnerVolumeSpecName "kube-api-access-c7zgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:55:32.465177 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.465129    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-podinfo" pod "7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" (UID: "7c524b88-93cc-4890-aca0-d1dbd4b2f2dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:55:32.465466 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.465444    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-token" pod "7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" (UID: "7c524b88-93cc-4890-aca0-d1dbd4b2f2dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:55:32.465583 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.465529    5428 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-envoy" pod "7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" (UID: "7c524b88-93cc-4890-aca0-d1dbd4b2f2dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:55:32.484351 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.484334    5428 scope.go:122] "RemoveContainer" containerID="aed1b6105d7d4e5f94879c29f415f77abebeaf508b5ea2f7419a6fc1080d94ef"
Oct 29 23:55:32.484627 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:55:32.484607    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aed1b6105d7d4e5f94879c29f415f77abebeaf508b5ea2f7419a6fc1080d94ef\": container with ID starting with aed1b6105d7d4e5f94879c29f415f77abebeaf508b5ea2f7419a6fc1080d94ef not found: ID does not exist" containerID="aed1b6105d7d4e5f94879c29f415f77abebeaf508b5ea2f7419a6fc1080d94ef"
Oct 29 23:55:32.484671 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.484639    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aed1b6105d7d4e5f94879c29f415f77abebeaf508b5ea2f7419a6fc1080d94ef"} err="failed to get container status \"aed1b6105d7d4e5f94879c29f415f77abebeaf508b5ea2f7419a6fc1080d94ef\": rpc error: code = NotFound desc = could not find container \"aed1b6105d7d4e5f94879c29f415f77abebeaf508b5ea2f7419a6fc1080d94ef\": container with ID starting with aed1b6105d7d4e5f94879c29f415f77abebeaf508b5ea2f7419a6fc1080d94ef not found: ID does not exist"
Oct 29 23:55:32.484671 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.484657    5428 scope.go:122] "RemoveContainer" containerID="122b37eccf0ed7fcb6d783ae4cb929f99107edc0f7a1a2e5babe95c59fd99637"
Oct 29 23:55:32.484941 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:55:32.484919    5428 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"122b37eccf0ed7fcb6d783ae4cb929f99107edc0f7a1a2e5babe95c59fd99637\": container with ID starting with 122b37eccf0ed7fcb6d783ae4cb929f99107edc0f7a1a2e5babe95c59fd99637 not found: ID does not exist" containerID="122b37eccf0ed7fcb6d783ae4cb929f99107edc0f7a1a2e5babe95c59fd99637"
Oct 29 23:55:32.485001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.484946    5428 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"122b37eccf0ed7fcb6d783ae4cb929f99107edc0f7a1a2e5babe95c59fd99637"} err="failed to get container status \"122b37eccf0ed7fcb6d783ae4cb929f99107edc0f7a1a2e5babe95c59fd99637\": rpc error: code = NotFound desc = could not find container \"122b37eccf0ed7fcb6d783ae4cb929f99107edc0f7a1a2e5babe95c59fd99637\": container with ID starting with 122b37eccf0ed7fcb6d783ae4cb929f99107edc0f7a1a2e5babe95c59fd99637 not found: ID does not exist"
Oct 29 23:55:32.564951 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.564929    5428 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-envoy\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:32.564951 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.564948    5428 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7zgm\" (UniqueName: \"kubernetes.io/projected/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-kube-api-access-c7zgm\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:32.565080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.564957    5428 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-token\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:32.565080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.564967    5428 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-podinfo\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:32.565080 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.564975    5428 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc-istio-data\") on node \"ip-10-0-134-127.ec2.internal\" DevicePath \"\""
Oct 29 23:55:32.800388 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.800357    5428 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"]
Oct 29 23:55:32.812447 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:32.812423    5428 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-99dfb0-openshift-default-85ffc958b7-n5xpp"]
Oct 29 23:55:34.393811 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:34.393753    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:34.775432 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:34.775359    5428 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" path="/var/lib/kubelet/pods/7c524b88-93cc-4890-aca0-d1dbd4b2f2dc/volumes"
Oct 29 23:55:35.070146 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:35.070053    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:35.456712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:35.456675    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:35.579058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:35.579021    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:36.489953 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:36.489921    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:39.464140 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:39.464110    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:42.088237 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:42.088202    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:55:42.445367 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:42.445342    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5k8zg_bf6c054d-59e2-43d6-bad1-ee8c54690e83/server/0.log"
Oct 29 23:55:42.643846 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:42.643815    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-86vff_27d64fc8-f24c-433d-a320-9d272d87bd8d/manager/0.log"
Oct 29 23:55:42.891058 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:42.891034    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_trustyai-service-operator-controller-manager-5997944679-kl2s2_36e122a4-d2b7-44d4-8a45-994ea1fb5af8/manager/0.log"
Oct 29 23:55:43.093360 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:43.093334    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vjnhg_408dc1af-76b7-41fa-b262-915fcf9b4ffa/proxy/0.log"
Oct 29 23:55:44.049235 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:44.049201    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f48949686-sngj6_96b9b7f3-d372-44ca-adfb-79546be9f85e/authorino/0.log"
Oct 29 23:55:47.367692 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:55:47.367646    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:55:54.561712 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:55:54.561678    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7bdc847df-4dfgz_6d23d603-f91c-4a7b-85e0-8c25b786b480/manager/0.log"
Oct 29 23:56:04.770116 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:04.770084    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lz8kp_1ad543b2-b749-40f3-9210-4026141bbf3d/manager/0.log"
Oct 29 23:56:04.981001 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:04.980964    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-6pqtr_6d668198-fafe-4276-9474-e55b192582bf/manager/0.log"
Oct 29 23:56:05.926530 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:05.926498    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5k8zg_bf6c054d-59e2-43d6-bad1-ee8c54690e83/server/0.log"
Oct 29 23:56:06.128338 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:06.128305    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-86vff_27d64fc8-f24c-433d-a320-9d272d87bd8d/manager/0.log"
Oct 29 23:56:06.362085 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:06.362012    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_trustyai-service-operator-controller-manager-5997944679-kl2s2_36e122a4-d2b7-44d4-8a45-994ea1fb5af8/manager/0.log"
Oct 29 23:56:06.549906 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:06.549876    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vjnhg_408dc1af-76b7-41fa-b262-915fcf9b4ffa/proxy/0.log"
Oct 29 23:56:08.121623 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:08.121587    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-42f63f53-ggszg_905a49ab-1b16-4584-b8b2-c6eb1e597134/revoke-keys/0.log"
Oct 29 23:56:08.294176 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:08.294145    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-57bb2a42-8pf75_84aa240d-42ed-43c5-a84b-001a707311ba/revoke-keys/0.log"
Oct 29 23:56:10.861218 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:10.861186    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f48949686-sngj6_96b9b7f3-d372-44ca-adfb-79546be9f85e/authorino/0.log"
Oct 29 23:56:11.088898 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:11.088853    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-dqxbf_7f6616af-b4f8-4cd9-9cd1-c356b28d97c4/manager/0.log"
Oct 29 23:56:13.591191 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:13.591161    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7_6f0b257f-c278-4918-a13a-284e1fd77231/istio-proxy/0.log"
Oct 29 23:56:14.163670 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:14.163646    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67f887f896-mv4wq_fafd9f2b-e714-4a9f-9ac9-20031efbf034/kube-auth-proxy/0.log"
Oct 29 23:56:17.367147 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:56:17.367108    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:56:24.542325 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:24.542271    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-lbp8m_fbea7545-5d53-4ba1-bb53-584acf2a2bc7/istio-proxy/1.log"
Oct 29 23:56:24.954983 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:24.954930    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zv5lm_8712a755-0297-44aa-8c53-361ff3b51ede/payload-pre-processing/0.log"
Oct 29 23:56:25.142127 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:25.142103    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-822k7_5d767414-e11d-4113-90ee-f2f0ae1588a1/payload-processing/0.log"
Oct 29 23:56:25.749096 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:25.749011    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-lbp8m_fbea7545-5d53-4ba1-bb53-584acf2a2bc7/istio-proxy/0.log"
Oct 29 23:56:26.302180 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:26.302149    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-f9gzc_87e30ba6-6bca-4971-844c-c9509f5b0d0b/main/0.log"
Oct 29 23:56:26.695669 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:26.695643    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-c2s6t_cbcb9ab9-9d59-4f78-a953-aa6138fb888a/main/0.log"
Oct 29 23:56:27.079338 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:27.079248    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8t8dbj_2a7ce9ea-305c-4510-86db-2a9a74021633/main/0.log"
Oct 29 23:56:27.485642 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:27.485600    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-92zsc_c4034d76-e0f2-4674-928d-657820813e88/main/0.log"
Oct 29 23:56:34.506773 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:34.506712    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:56:36.258535 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:36.258492    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:56:47.366666 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:56:47.366627    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:56:53.805155 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:53.805118    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:56:57.821347 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:57.821261    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:56:57.947958 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:56:57.947927    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:57:04.021423 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:04.021391    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qmfpg_6edb8e5f-e185-48be-8fed-5ac6899134fa/global-pull-secret-syncer/0.log"
Oct 29 23:57:04.759854 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:04.759821    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log"
Oct 29 23:57:05.177634 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:05.177606    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-127.ec2.internal_a47eb3de0ec2692f9f0b4d5a8aa019c1/haproxy/0.log"
Oct 29 23:57:12.056387 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:12.056356    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f48949686-sngj6_96b9b7f3-d372-44ca-adfb-79546be9f85e/authorino/0.log"
Oct 29 23:57:12.164066 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:12.164036    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-dqxbf_7f6616af-b4f8-4cd9-9cd1-c356b28d97c4/manager/0.log"
Oct 29 23:57:15.996149 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:15.996118    5428 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"]
Oct 29 23:57:15.996573 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:15.996520    5428 cpu_manager.go:414] "containerMap: removing container" podUID="7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" containerName="istio-proxy"
Oct 29 23:57:15.996573 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:15.996532    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" containerName="istio-proxy"
Oct 29 23:57:15.996573 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:15.996543    5428 cpu_manager.go:414] "containerMap: removing container" podUID="2032f74c-64e7-4c12-a969-c5b7eb33eb53" containerName="payload-pre-processing"
Oct 29 23:57:15.996573 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:15.996548    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2032f74c-64e7-4c12-a969-c5b7eb33eb53" containerName="payload-pre-processing"
Oct 29 23:57:15.996573 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:15.996561    5428 cpu_manager.go:414] "containerMap: removing container" podUID="c04a4a59-e364-41e3-bf19-cb3534ad9ec8" containerName="payload-processing"
Oct 29 23:57:15.996573 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:15.996566    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04a4a59-e364-41e3-bf19-cb3534ad9ec8" containerName="payload-processing"
Oct 29 23:57:15.996794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:15.996582    5428 cpu_manager.go:414] "containerMap: removing container" podUID="7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" containerName="istio-proxy"
Oct 29 23:57:15.996794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:15.996587    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" containerName="istio-proxy"
Oct 29 23:57:15.996794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:15.996593    5428 cpu_manager.go:414] "containerMap: removing container" podUID="905a49ab-1b16-4584-b8b2-c6eb1e597134" containerName="revoke-keys"
Oct 29 23:57:15.996794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:15.996598    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="905a49ab-1b16-4584-b8b2-c6eb1e597134" containerName="revoke-keys"
Oct 29 23:57:15.996794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:15.996614    5428 cpu_manager.go:414] "containerMap: removing container" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main"
Oct 29 23:57:15.996794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:15.996619    5428 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main"
Oct 29 23:57:15.996794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:15.996696    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c62dd36-e795-4778-902e-a990d1998123" containerName="main"
Oct 29 23:57:15.996794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:15.996711    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c524b88-93cc-4890-aca0-d1dbd4b2f2dc" containerName="istio-proxy"
Oct 29 23:57:15.996794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:15.996719    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="2032f74c-64e7-4c12-a969-c5b7eb33eb53" containerName="payload-pre-processing"
Oct 29 23:57:15.996794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:15.996728    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04a4a59-e364-41e3-bf19-cb3534ad9ec8" containerName="payload-processing"
Oct 29 23:57:15.996794 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:15.996740    5428 memory_manager.go:365] "RemoveStaleState removing state" podUID="905a49ab-1b16-4584-b8b2-c6eb1e597134" containerName="revoke-keys"
Oct 29 23:57:16.000037 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.000016    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:16.015057 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.015026    5428 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t99pv\"/\"default-dockercfg-8lgdb\""
Oct 29 23:57:16.017014 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.016987    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"]
Oct 29 23:57:16.097938 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.097904    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c55966fe-ec38-4ffd-98c8-73b73f1a2edd-sys\") pod \"perf-node-gather-daemonset-wp444\" (UID: \"c55966fe-ec38-4ffd-98c8-73b73f1a2edd\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:16.098142 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.097956    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c55966fe-ec38-4ffd-98c8-73b73f1a2edd-proc\") pod \"perf-node-gather-daemonset-wp444\" (UID: \"c55966fe-ec38-4ffd-98c8-73b73f1a2edd\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:16.098142 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.098102    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c55966fe-ec38-4ffd-98c8-73b73f1a2edd-lib-modules\") pod \"perf-node-gather-daemonset-wp444\" (UID: \"c55966fe-ec38-4ffd-98c8-73b73f1a2edd\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:16.098259 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.098148    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c55966fe-ec38-4ffd-98c8-73b73f1a2edd-podres\") pod \"perf-node-gather-daemonset-wp444\" (UID: \"c55966fe-ec38-4ffd-98c8-73b73f1a2edd\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:16.098297 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.098248    5428 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tf5w\" (UniqueName: \"kubernetes.io/projected/c55966fe-ec38-4ffd-98c8-73b73f1a2edd-kube-api-access-2tf5w\") pod \"perf-node-gather-daemonset-wp444\" (UID: \"c55966fe-ec38-4ffd-98c8-73b73f1a2edd\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:16.106063 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.106040    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t99pv\"/\"openshift-service-ca.crt\""
Oct 29 23:57:16.106173 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.106073    5428 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t99pv\"/\"kube-root-ca.crt\""
Oct 29 23:57:16.199023 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.198994    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c55966fe-ec38-4ffd-98c8-73b73f1a2edd-proc\") pod \"perf-node-gather-daemonset-wp444\" (UID: \"c55966fe-ec38-4ffd-98c8-73b73f1a2edd\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:16.199216 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.199075    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c55966fe-ec38-4ffd-98c8-73b73f1a2edd-lib-modules\") pod \"perf-node-gather-daemonset-wp444\" (UID: \"c55966fe-ec38-4ffd-98c8-73b73f1a2edd\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:16.199216 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.199105    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c55966fe-ec38-4ffd-98c8-73b73f1a2edd-podres\") pod \"perf-node-gather-daemonset-wp444\" (UID: \"c55966fe-ec38-4ffd-98c8-73b73f1a2edd\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:16.199216 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.199133    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tf5w\" (UniqueName: \"kubernetes.io/projected/c55966fe-ec38-4ffd-98c8-73b73f1a2edd-kube-api-access-2tf5w\") pod \"perf-node-gather-daemonset-wp444\" (UID: \"c55966fe-ec38-4ffd-98c8-73b73f1a2edd\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:16.199216 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.199162    5428 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c55966fe-ec38-4ffd-98c8-73b73f1a2edd-sys\") pod \"perf-node-gather-daemonset-wp444\" (UID: \"c55966fe-ec38-4ffd-98c8-73b73f1a2edd\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:16.200296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.200269    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c55966fe-ec38-4ffd-98c8-73b73f1a2edd-lib-modules\") pod \"perf-node-gather-daemonset-wp444\" (UID: \"c55966fe-ec38-4ffd-98c8-73b73f1a2edd\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:16.200296 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.200287    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c55966fe-ec38-4ffd-98c8-73b73f1a2edd-proc\") pod \"perf-node-gather-daemonset-wp444\" (UID: \"c55966fe-ec38-4ffd-98c8-73b73f1a2edd\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:16.200481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.200273    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c55966fe-ec38-4ffd-98c8-73b73f1a2edd-sys\") pod \"perf-node-gather-daemonset-wp444\" (UID: \"c55966fe-ec38-4ffd-98c8-73b73f1a2edd\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:16.200481 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.200272    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c55966fe-ec38-4ffd-98c8-73b73f1a2edd-podres\") pod \"perf-node-gather-daemonset-wp444\" (UID: \"c55966fe-ec38-4ffd-98c8-73b73f1a2edd\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:16.230792 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.230764    5428 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tf5w\" (UniqueName: \"kubernetes.io/projected/c55966fe-ec38-4ffd-98c8-73b73f1a2edd-kube-api-access-2tf5w\") pod \"perf-node-gather-daemonset-wp444\" (UID: \"c55966fe-ec38-4ffd-98c8-73b73f1a2edd\") " pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:16.312926 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.312855    5428 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:16.564199 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.564118    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7bdc847df-4dfgz_6d23d603-f91c-4a7b-85e0-8c25b786b480/manager/0.log"
Oct 29 23:57:16.615235 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.615206    5428 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"]
Oct 29 23:57:16.618493 ip-10-0-134-127 kubenswrapper[5428]: W1029 23:57:16.618455    5428 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc55966fe_ec38_4ffd_98c8_73b73f1a2edd.slice/crio-8596c333279020ad3d6af3279916700688f132dc3862a9314577540a13be638b WatchSource:0}: Error finding container 8596c333279020ad3d6af3279916700688f132dc3862a9314577540a13be638b: Status 404 returned error can't find the container with id 8596c333279020ad3d6af3279916700688f132dc3862a9314577540a13be638b
Oct 29 23:57:16.639642 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.639619    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lz8kp_1ad543b2-b749-40f3-9210-4026141bbf3d/manager/0.log"
Oct 29 23:57:16.770438 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.770406    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-6pqtr_6d668198-fafe-4276-9474-e55b192582bf/manager/0.log"
Oct 29 23:57:16.809378 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.809350    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7bdc847df-4dfgz_6d23d603-f91c-4a7b-85e0-8c25b786b480/manager/0.log"
Oct 29 23:57:16.857650 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.857567    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lz8kp_1ad543b2-b749-40f3-9210-4026141bbf3d/manager/0.log"
Oct 29 23:57:16.961574 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.961546    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-6pqtr_6d668198-fafe-4276-9474-e55b192582bf/manager/0.log"
Oct 29 23:57:16.996089 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.996046    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444" event={"ID":"c55966fe-ec38-4ffd-98c8-73b73f1a2edd","Type":"ContainerStarted","Data":"7d875d9c5f63b0421a5b1e86d704dccdd7ff4aca9725c6b0046630414995c144"}
Oct 29 23:57:16.996538 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.996502    5428 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444" event={"ID":"c55966fe-ec38-4ffd-98c8-73b73f1a2edd","Type":"ContainerStarted","Data":"8596c333279020ad3d6af3279916700688f132dc3862a9314577540a13be638b"}
Oct 29 23:57:16.996949 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.996579    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:16.999325 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:16.999302    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5pzlf_c44b4b5f-8aa2-4cf2-b72a-16b5002260b6/node-exporter/0.log"
Oct 29 23:57:17.037207 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:17.037156    5428 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444" podStartSLOduration=2.037143093 podStartE2EDuration="2.037143093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:57:17.035488841 +0000 UTC m=+4440.378437587" watchObservedRunningTime="2026-10-29 23:57:17.037143093 +0000 UTC m=+4440.380091860"
Oct 29 23:57:17.118246 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:17.118207    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5pzlf_c44b4b5f-8aa2-4cf2-b72a-16b5002260b6/kube-rbac-proxy/0.log"
Oct 29 23:57:17.241083 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:17.241053    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5pzlf_c44b4b5f-8aa2-4cf2-b72a-16b5002260b6/init-textfile/0.log"
Oct 29 23:57:17.369524 ip-10-0-134-127 kubenswrapper[5428]: E1029 23:57:17.369413    5428 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gpthh" containerName="extractor"
Oct 29 23:57:18.472939 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:18.472903    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40fb2dc7-336c-4483-93e7-68309e345b22/prometheus/0.log"
Oct 29 23:57:18.590303 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:18.590270    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40fb2dc7-336c-4483-93e7-68309e345b22/config-reloader/0.log"
Oct 29 23:57:18.709781 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:18.709754    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40fb2dc7-336c-4483-93e7-68309e345b22/thanos-sidecar/0.log"
Oct 29 23:57:18.828100 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:18.828023    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40fb2dc7-336c-4483-93e7-68309e345b22/kube-rbac-proxy-web/0.log"
Oct 29 23:57:18.947088 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:18.947057    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40fb2dc7-336c-4483-93e7-68309e345b22/kube-rbac-proxy/0.log"
Oct 29 23:57:19.005003 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:19.004974    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5k8zg_bf6c054d-59e2-43d6-bad1-ee8c54690e83/server/0.log"
Oct 29 23:57:19.068200 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:19.068174    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40fb2dc7-336c-4483-93e7-68309e345b22/kube-rbac-proxy-thanos/0.log"
Oct 29 23:57:19.199472 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:19.199447    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40fb2dc7-336c-4483-93e7-68309e345b22/init-config-reloader/0.log"
Oct 29 23:57:19.212423 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:19.212401    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5k8zg_bf6c054d-59e2-43d6-bad1-ee8c54690e83/server/0.log"
Oct 29 23:57:19.339636 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:19.339582    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-87hkr_38dfb740-4469-4101-a309-4070aeb2dfe3/prometheus-operator/0.log"
Oct 29 23:57:19.472937 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:19.472860    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-87hkr_38dfb740-4469-4101-a309-4070aeb2dfe3/kube-rbac-proxy/0.log"
Oct 29 23:57:19.905436 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:19.905370    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-86vff_27d64fc8-f24c-433d-a320-9d272d87bd8d/manager/0.log"
Oct 29 23:57:19.940580 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:19.940515    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_trustyai-service-operator-controller-manager-5997944679-kl2s2_36e122a4-d2b7-44d4-8a45-994ea1fb5af8/manager/0.log"
Oct 29 23:57:20.014096 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:20.014045    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vjnhg_408dc1af-76b7-41fa-b262-915fcf9b4ffa/proxy/0.log"
Oct 29 23:57:20.077824 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:20.077778    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-86vff_27d64fc8-f24c-433d-a320-9d272d87bd8d/manager/0.log"
Oct 29 23:57:20.110167 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:20.110130    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_trustyai-service-operator-controller-manager-5997944679-kl2s2_36e122a4-d2b7-44d4-8a45-994ea1fb5af8/manager/0.log"
Oct 29 23:57:20.175276 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:20.175203    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vjnhg_408dc1af-76b7-41fa-b262-915fcf9b4ffa/proxy/0.log"
Oct 29 23:57:20.443591 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:20.443502    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:57:21.687231 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:21.687176    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7bdc847df-4dfgz_6d23d603-f91c-4a7b-85e0-8c25b786b480/manager/0.log"
Oct 29 23:57:21.756553 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:21.756522    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lz8kp_1ad543b2-b749-40f3-9210-4026141bbf3d/manager/0.log"
Oct 29 23:57:21.762288 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:21.762265    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7bdc847df-4dfgz_6d23d603-f91c-4a7b-85e0-8c25b786b480/manager/0.log"
Oct 29 23:57:21.802701 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:21.802677    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lz8kp_1ad543b2-b749-40f3-9210-4026141bbf3d/manager/0.log"
Oct 29 23:57:21.861038 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:21.861012    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-6pqtr_6d668198-fafe-4276-9474-e55b192582bf/manager/0.log"
Oct 29 23:57:21.905677 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:21.905651    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-6pqtr_6d668198-fafe-4276-9474-e55b192582bf/manager/0.log"
Oct 29 23:57:23.011212 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:23.011187    5428 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t99pv/perf-node-gather-daemonset-wp444"
Oct 29 23:57:24.372883 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:24.372851    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5k8zg_bf6c054d-59e2-43d6-bad1-ee8c54690e83/server/0.log"
Oct 29 23:57:24.419283 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:24.419255    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5k8zg_bf6c054d-59e2-43d6-bad1-ee8c54690e83/server/0.log"
Oct 29 23:57:24.601482 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:24.601448    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8494bcdf8c-w5qzf_360d79bf-6e2e-4060-8318-f0d86c71c2e8/console/0.log"
Oct 29 23:57:25.270455 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:25.270418    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-86vff_27d64fc8-f24c-433d-a320-9d272d87bd8d/manager/0.log"
Oct 29 23:57:25.339689 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:25.339654    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_trustyai-service-operator-controller-manager-5997944679-kl2s2_36e122a4-d2b7-44d4-8a45-994ea1fb5af8/manager/0.log"
Oct 29 23:57:25.349822 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:25.349780    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-86vff_27d64fc8-f24c-433d-a320-9d272d87bd8d/manager/0.log"
Oct 29 23:57:25.409549 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:25.409522    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vjnhg_408dc1af-76b7-41fa-b262-915fcf9b4ffa/proxy/0.log"
Oct 29 23:57:25.409941 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:25.409628    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_trustyai-service-operator-controller-manager-5997944679-kl2s2_36e122a4-d2b7-44d4-8a45-994ea1fb5af8/manager/0.log"
Oct 29 23:57:25.485354 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:25.485323    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vjnhg_408dc1af-76b7-41fa-b262-915fcf9b4ffa/proxy/0.log"
Oct 29 23:57:27.362434 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:27.362407    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7bdc847df-4dfgz_6d23d603-f91c-4a7b-85e0-8c25b786b480/manager/0.log"
Oct 29 23:57:27.403702 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:27.403670    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lz8kp_1ad543b2-b749-40f3-9210-4026141bbf3d/manager/0.log"
Oct 29 23:57:27.462734 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:27.462708    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7bdc847df-4dfgz_6d23d603-f91c-4a7b-85e0-8c25b786b480/manager/0.log"
Oct 29 23:57:27.513610 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:27.513572    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lz8kp_1ad543b2-b749-40f3-9210-4026141bbf3d/manager/0.log"
Oct 29 23:57:27.516716 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:27.516692    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-6pqtr_6d668198-fafe-4276-9474-e55b192582bf/manager/0.log"
Oct 29 23:57:27.615894 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:27.615867    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-6pqtr_6d668198-fafe-4276-9474-e55b192582bf/manager/0.log"
Oct 29 23:57:28.948159 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:28.948133    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dcx8f_ca99b14b-c183-4966-85c2-b685be64e4cb/dns/0.log"
Oct 29 23:57:29.063400 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:29.063372    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dcx8f_ca99b14b-c183-4966-85c2-b685be64e4cb/kube-rbac-proxy/0.log"
Oct 29 23:57:29.927639 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:29.927600    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q8pg5_e50192cd-cb7b-4b3e-978f-661093207b81/dns-node-resolver/0.log"
Oct 29 23:57:30.043574 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:30.043546    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5k8zg_bf6c054d-59e2-43d6-bad1-ee8c54690e83/server/0.log"
Oct 29 23:57:30.127124 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:30.127095    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5k8zg_bf6c054d-59e2-43d6-bad1-ee8c54690e83/server/0.log"
Oct 29 23:57:30.921526 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:30.921487    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-86vff_27d64fc8-f24c-433d-a320-9d272d87bd8d/manager/0.log"
Oct 29 23:57:30.983512 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:30.983486    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_trustyai-service-operator-controller-manager-5997944679-kl2s2_36e122a4-d2b7-44d4-8a45-994ea1fb5af8/manager/0.log"
Oct 29 23:57:31.037680 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:31.037644    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-86vff_27d64fc8-f24c-433d-a320-9d272d87bd8d/manager/0.log"
Oct 29 23:57:31.063187 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:31.063156    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vjnhg_408dc1af-76b7-41fa-b262-915fcf9b4ffa/proxy/0.log"
Oct 29 23:57:31.101564 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:31.101525    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_trustyai-service-operator-controller-manager-5997944679-kl2s2_36e122a4-d2b7-44d4-8a45-994ea1fb5af8/manager/0.log"
Oct 29 23:57:31.169951 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:31.169916    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vjnhg_408dc1af-76b7-41fa-b262-915fcf9b4ffa/proxy/0.log"
Oct 29 23:57:31.449943 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:31.449913    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n27f7_da8029d1-50de-4612-adce-3a3f582c74d8/node-ca/0.log"
Oct 29 23:57:31.865136 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:31.865109    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7_6f0b257f-c278-4918-a13a-284e1fd77231/istio-proxy/0.log"
Oct 29 23:57:33.039649 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:33.039616    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7_6f0b257f-c278-4918-a13a-284e1fd77231/istio-proxy/0.log"
Oct 29 23:57:33.143896 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:33.143866    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7_6f0b257f-c278-4918-a13a-284e1fd77231/istio-proxy/0.log"
Oct 29 23:57:33.633007 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:33.632980    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-bb66448f7-kzdk7_6f0b257f-c278-4918-a13a-284e1fd77231/istio-proxy/0.log"
Oct 29 23:57:34.008013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:34.007933    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67f887f896-mv4wq_fafd9f2b-e714-4a9f-9ac9-20031efbf034/kube-auth-proxy/0.log"
Oct 29 23:57:34.136009 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:34.135981    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-lbp8m_fbea7545-5d53-4ba1-bb53-584acf2a2bc7/istio-proxy/0.log"
Oct 29 23:57:34.137026 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:34.137003    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-lbp8m_fbea7545-5d53-4ba1-bb53-584acf2a2bc7/istio-proxy/1.log"
Oct 29 23:57:34.251013 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:34.250979    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zv5lm_8712a755-0297-44aa-8c53-361ff3b51ede/payload-pre-processing/0.log"
Oct 29 23:57:34.311602 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:34.311522    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-822k7_5d767414-e11d-4113-90ee-f2f0ae1588a1/payload-processing/0.log"
Oct 29 23:57:34.642868 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:34.642835    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:57:35.618446 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:35.618411    5428 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jn2n_fea00aae-5dd9-48ef-b7d4-af08eed1481d/konnectivity-agent/0.log" limit=1024
Oct 29 23:57:35.985913 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:35.985834    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67f887f896-mv4wq_fafd9f2b-e714-4a9f-9ac9-20031efbf034/kube-auth-proxy/0.log"
Oct 29 23:57:36.115205 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:36.115175    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67f887f896-mv4wq_fafd9f2b-e714-4a9f-9ac9-20031efbf034/kube-auth-proxy/0.log"
Oct 29 23:57:36.122590 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:36.122561    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-lbp8m_fbea7545-5d53-4ba1-bb53-584acf2a2bc7/istio-proxy/0.log"
Oct 29 23:57:36.123665 ip-10-0-134-127 kubenswrapper[5428]: I1029 23:57:36.123639    5428 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-lbp8m_fbea7545-5d53-4ba1-bb53-584acf2a2bc7/istio-proxy/1.log"